About the job
Dice is the leading career destination for tech experts at every stage of their careers. Our client, Thoughtwave Software and Solutions, is seeking the following. Apply via Dice today!
Role: Data Engineer
Duration: 6-month contract to hire
Location: Remote
Visa: W2 role
Required Skills:
Data Pipeline and Storage: Detailed the process of collecting logs from various sources, structuring them through a pipeline using vectored dev for rate limiting and security, and storing them in ClickHouse as the database.
Querying and Visualization: They plan to offer Grafana for standard querying but will support other tools like Databricks, Power BI, and Excel. An open-source tool called Keep will serve as the middle layer for event management and rules engine.
Event Management: Keep will query the lake for triggering events in ServiceNow, acting as an intermediary due to ServiceNow's inability to handle millions of rules directly.
Scale and Talent Needs: The platform aims to handle 100 petabytes a month initially, requiring sharp talent, especially in ClickHouse and vectored dev. They are seeking vendor support for architecture and operational expertise.
Challenges and Solutions: The discussion also covered the challenges of scaling, the need for a multi-tenant solution, and the importance of optimizing SQL queries for efficiency. The goal is to consolidate various data sources into a single, queryable database to facilitate correlation and analysis.
--
Thanks & Regards,
Kalyan Sadanala| Talent acquisition specialist
Thoughtwave Software and Solutions
314 N. Lake St, Suite 6, Aurora IL 60506
Desk: , EXTN: 148
Text Message:
Email:
Website:
Linked in:
Role: Data Engineer
Duration: 6-month contract to hire
Location: Remote
Visa: W2 role
Required Skills:
- ClickHouse- 2+ Years 2. Kubernetes - understanding infra. 3. AWS
Data Pipeline and Storage: Detailed the process of collecting logs from various sources, structuring them through a pipeline using vectored dev for rate limiting and security, and storing them in ClickHouse as the database.
Querying and Visualization: They plan to offer Grafana for standard querying but will support other tools like Databricks, Power BI, and Excel. An open-source tool called Keep will serve as the middle layer for event management and rules engine.
Event Management: Keep will query the lake for triggering events in ServiceNow, acting as an intermediary due to ServiceNow's inability to handle millions of rules directly.
Scale and Talent Needs: The platform aims to handle 100 petabytes a month initially, requiring sharp talent, especially in ClickHouse and vectored dev. They are seeking vendor support for architecture and operational expertise.
Challenges and Solutions: The discussion also covered the challenges of scaling, the need for a multi-tenant solution, and the importance of optimizing SQL queries for efficiency. The goal is to consolidate various data sources into a single, queryable database to facilitate correlation and analysis.
--
Thanks & Regards,
Kalyan Sadanala| Talent acquisition specialist
Thoughtwave Software and Solutions
314 N. Lake St, Suite 6, Aurora IL 60506
Desk: , EXTN: 148
Text Message:
Email:
Website:
Linked in:
