About the job
Role: Data Engineer
Duration: 6-month contract to hire
Location: Remote
Visa: W2 role
Required Skills:
- ClickHouse- 2+ Years 2. Kubernetes - understanding infra. 3. AWS
Data Pipeline and Storage: Detailed the process of collecting logs from various sources, structuring them through a pipeline using vectored dev for rate limiting and security, and storing them in ClickHouse as the database.
Querying and Visualization: They plan to offer Grafana for standard querying but will support other tools like Databricks, Power BI, and Excel. An open-source tool called Keep will serve as the middle layer for event management and rules engine.
Event Management: Keep will query the lake for triggering events in ServiceNow, acting as an intermediary due to ServiceNow's inability to handle millions of rules directly.
Scale and Talent Needs: The platform aims to handle 100 petabytes a month initially, requiring sharp talent, especially in ClickHouse and vectored dev. They are seeking vendor support for architecture and operational expertise.
Challenges and Solutions: The discussion also covered the challenges of scaling, the need for a multi-tenant solution, and the importance of optimizing SQL queries for efficiency. The goal is to consolidate various data sources into a single, queryable database to facilitate correlation and analysis.
--
Thanks & Regards,
Kalyan Sadanala| Talent acquisition specialist
Thoughtwave Software and Solutions
314 N. Lake St, Suite 6, Aurora IL 60506
Desk: , EXTN: 148
Text Message:
Email:
Website:
Linked in: