“Wonderland” for Data Engineers at FSOFT

Talking about to the projects at FHN that are closely related to data processing, it is impossible not to mention the S-Data Pilots project with an American client and 7 talented Data Engineers (DEs). DEs are continuing to train, hone and enhance their expertise to be able to join any unit’s ‘battle’ when needed.

Join the battle in potential projects

As the unit responsible for building data capacity of FHN, FHN.BU91, under the leadership of Mr. Tran Phuc Khanh, is running many projects specializing in data processing, including S – Data Pilots.

S – Data Pilots is the result of cooperation between FPT, INET (Intellinet), and client S – one of the largest manufacturing companies in the world for floor-related products with an annual turnover of 6 billion USD, and 22,000 employees worldwide. Currently, the project is in the pilot phase, lasting 3 months.

With the task of solving data problems for the client, the project currently has 7 DEs, of which 3 are transferred from Dev. The rest are DEs who own international certificates and have experienced many projects as Data Engineers in FSOFT as well as other big companies.

Sharing specific problems in this project, PM Nguyen Van Toan said that owning an internal data set such as timesheet, salary, bonus, and personnel, client S needs to exploit a huge volume of data accumulated daily. Thereby exploiting, extracting, and finding useful insights and visualizing these insights on the dashboard is of vital importance. To do this, S hired FPT Intellinet to consult and deploy the package. Intellinet has “shaken hands” with the offshore team FHN.BU91 to run the project together.

In the S – Data Pilots project, DEs are in charge of two main areas of work, KPI Validation and Build data pipelines. In it, with KPI Validation, S will provide a window virtual machine that the team will access through the Citrix app. From this virtual machine, DE will use SQL Query data in all databases such as timesheet, salary, bonus, personnel, etc. to provide statistics. Then, DE uses Pivot, Query on MS excel, or BI/Tableau to actualize and summarize these numbers into charts or tables like reports exported to PDF files of the client. Then, DEs will check whether the numbers in the chart or table match the numbers in the report. Once a match is noted, the DE will list the tables used in the SQL statement to generate the reports in turn and use them in the data pipeline task later.

As for Build data pipelines, S needs a centralized data store for data coming from multiple sources. FPT Intellinet advised S to use Azure data lake. At this step, the DE will build data pipelines to get data from the tables in the KPI validation section of the respective databases to push them to the Azure data lake. The DEs will then clean the data and convert the data in a format to data warehouses via Azure bricks. All these data will be put by Data Scientists into AI algorithms, Data mining, etc. to get valuable insights that are hidden deep under these data and put them on the dashboard, supporting the Board of Directors or departments to make decisions in business – delivery.

Continually improve professional skills  

Commenting on the DEs at FSOFT, Mr. Toan said that FHN in particular and FSOFT, in general, are “thirsty” for them. However, it is difficult to fill the number of DEs in projects overnight because very few qualified engineers can participate in the playing field because it is demanding and requires strict requirements for foreign languages ​​and expertise especially when the Data Engineering industry in Vietnam is quite new.

Faced with this situation, FHN.BU91 has brought almost all data engineers and programmers wishing to switch to DE to enroll in the Udacity DE nano-degree course sponsored by FSOFT BOD. Engineers will be assigned to batches respectively in July, September 2021, and January 2022 to complete this course. This is one of the best DE courses in the world today that lasts for 5 months (5 to 10 hours/week).

Through this course, data engineers will understand the specific roles and responsibilities of a DE in the project and practice a real project building the data pipeline from start to finish on AWS. With this course, DEs can easily access any data project. In addition, FHN. BU91 is conducting sharing sessions on Data Engineering, and at the same time organizing CoEs to advise, support, and train programmers who want to switch to Data Engineering.

QueDT