Sketch2Code, a Microsoft AI-powered tool, creates webpages and applications from drawings

Microsoft has developed a web design tool that uses artificial intelligence to translate website designs into workable HTML code.

According to Microsoft, AI’s senior product manager Tara Shankar Jana, the tool, branded Sketch2Code, would “motivate every developer and every organization to do more with AI.”

It emerged from the “inherent” difficulty of sending a designer an image of a wire or app design from a whiteboard or piece of paper so that they could create HTML mockups.

Microsoft designed a web-based application to streamline this process and eliminate the need for additional human participation.

Instead, photographs of drawings are uploaded to AI servers on Microsoft’s Azure cloud infrastructure.

Apps that allow you to drag-and-drop items to build a website are nothing new.

Many companies provide services for converting original ideas into digital workspaces, but this is the first to use artificial intelligence to complete the process.

Sketch2Code’s AI compares images to a pre-built AI system, which creates an HTML code foundation and ultimately a full app.

At the heart of this technology is a “custom vision object prediction model,” which is essentially an image recognition tool designed specifically to use data from hand-drawn images.

The model recognizes basic HTML elements like buttons, labels, and text boxes, allowing it to predict their existence in any picture.

It can even recognize handwritten text in the boxes and turn it into a fully working app or webpage.

The tool is available for developers on GitHub, and its code is independent of HTML and can be extracted using XMAL and Universal Windows Platform, according to Shankar Jana.

Will Sketch2Code put an end to website design and development? Certainly not!

On paper, Sketch2Code appears to have a significant benefit, but in fact, companies are not searching for AI code they do not know how to edit or fix. This does not mean, however, that website designs should not be inventive and adapt to new technology.

Sketch2Code has been under development for a long time, and it’s only a matter of time before Microsoft’s innovative AI picks up new tricks. Because Sketch2Code is based on an AI model, it will begin creating higher-quality code from the human-designed samples it has access to.

The Method

  1. The user must first sketch a hand-drawn prototype, that is then uploaded to the site.
  2. The proprietary vision algorithm would then forecast the HTML components and their locations on the image in the following phase.
  3. A layout method uses the spatial information of the bounding boxes of the items expected to produce a grid structure that contains them all.
  4. The HTML generating process takes all the data they provide it and create HTML markup code that reflects the outcome.
  5. It takes some time, depending on how simple the design is, and then the user may get the HTML code.

The Process

Sketch2Code recognizes the design patterns given by the user at first. It analyzes the handwritten text, draws the structures, and then generates HTML from the best match. This technology can identify what a user interface/user experience designer has drawn on a whiteboard and translate that information into the HTML code. A hand-drawn image may now create HTML wireframes.

Continuation of the previous Ink To Code project

Sketch2Code appears to be an improved version of Ink to Code, which claimed to develop a UI prototype, even if the user just had minimal coding knowledge. 

Sketch2Code is more advanced and versatile than Ink to Code. As an example, users had to sketch their ideas using a proprietary Windows 10 application, then paint iconography to represent text fields and other elements. 

Sketch2Code only requires the developer’s scrawled image to be uploaded to the site, and the AI performs the rest. If you want to experiment with Sketch2Code, you can get the source code on GitHub.

This isn’t the first time someone has tried to make low-code or no-code tools

Microsoft has invested a lot of time and effort into developing no-code and low-code technologies that allow users to build apps and websites with little or no coding knowledge.

PowerApps, a low-code programming platform introduced by Microsoft, allows local developers to construct applications by just dragging and dropping elements.

Microsoft is the only computer major to have launched such platforms; however, some companies, such as Google App Makers, have offered a low-code creation platform for customized business apps.


Various viewpoints contribute to the overall picture of the issue. For a worker, such a scenario might make work easier, but it could also result in you losing your job, therefore there is a constant dread of AI among workers. Sketch2Code, on the other side, may elicit a neutral response from someone who has previously used Microsoft services.

A user who thinks the app might be improved. Stated, it appears to be pretty fascinating; unfortunately, when I tested it with my layout, I could not get what I desired.

That’s why I feel it may use some fine-tuning.

It’s interesting how things work out, according to another user. Microsoft Expression Blend, for instance, had a XAML-based variation of this concept, but Microsoft discontinued Expression and Silverlight, as well as the Expression Web tools; however, Microsoft should have kept part of that stuff because it was great.

Social Media Marketing Can Help Your Businesses