Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions src/shared/inc/stringshared.h
Original file line number Diff line number Diff line change
Expand Up @@ -825,6 +825,30 @@ struct CaseInsensitiveCompare
}
};

inline std::wstring FormatBytes(uint64_t bytes)
{
Comment on lines +828 to +829
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if we could move this to wsl::windows::common::string and use the windows StrFormatByteSizeEx. It does have a slightly different oputput. StrFormatByteSizeEx uses 1024-based math with the Windows-conventional labels (KB, MB, GB, TB) — i.e. 1.00 KB means 1024 bytes. The current helper is 1000-based. So switching will change the numbers users see during pulls which would diverge from docker's behavior. However, it does give localized decimal seperator. I could see this going either way.

ALSO, if we are making a helper for this, we should probably get rid of the other FormatBytes method in ContainerTasks.cpp:37 and just use a common helper.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opportunity to centralize this, such as moving the one from ContainerTasks into stringshared.h

I'm not sure if Craig's PR is the right place to do that consolidation, but at the very least using the same method for it is appropriate.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to leave this PR as small as possible, so I will refrain for now from doing refactoring changes. If it blocks this PR from going in then I can do it!

constexpr double c_kB = 1000.0;
constexpr double c_MB = 1000.0 * 1000.0;
constexpr double c_GB = 1000.0 * 1000.0 * 1000.0;

Comment on lines +828 to +833
if (bytes >= static_cast<uint64_t>(c_GB))
{
return std::format(L"{:.2f} GB", bytes / c_GB);
}
else if (bytes >= static_cast<uint64_t>(c_MB))
{
return std::format(L"{:.2f} MB", bytes / c_MB);
}
else if (bytes >= static_cast<uint64_t>(c_kB))
{
return std::format(L"{:.2f} KB", bytes / c_kB);
Comment thread
craigloewen-msft marked this conversation as resolved.
Comment thread
craigloewen-msft marked this conversation as resolved.
Comment thread
craigloewen-msft marked this conversation as resolved.
}
else
{
return std::format(L"{} B", bytes);
}
}
Comment thread
craigloewen-msft marked this conversation as resolved.

} // namespace wsl::shared::string

template <>
Expand Down
4 changes: 3 additions & 1 deletion src/windows/inc/docker_schema.h
Original file line number Diff line number Diff line change
Expand Up @@ -584,8 +584,10 @@ struct BuildKitStatus
{
std::string id;
std::string vertex;
int64_t current{};
int64_t total{};

NLOHMANN_DEFINE_TYPE_INTRUSIVE_WITH_DEFAULT(BuildKitStatus, id, vertex);
NLOHMANN_DEFINE_TYPE_INTRUSIVE_WITH_DEFAULT(BuildKitStatus, id, vertex, current, total);
};

struct BuildKitLog
Expand Down
140 changes: 87 additions & 53 deletions src/windows/wslc/services/BuildImageCallback.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,10 @@ void BuildImageCallback::CollapseWindow()

m_lines.clear();
m_pendingLine.clear();
m_pullLines.clear();
}

HRESULT BuildImageCallback::OnProgress(LPCSTR status, LPCSTR id, ULONGLONG /*current*/, ULONGLONG /*total*/)
HRESULT BuildImageCallback::OnProgress(LPCSTR status, LPCSTR id, ULONGLONG current, ULONGLONG total)
try
{
if (status == nullptr || *status == '\0')
Expand All @@ -81,78 +82,103 @@ try
return S_OK;
}

const std::string_view idView = (id != nullptr) ? id : std::string_view{};
const bool isLog = (idView == "log");
const bool isPullProgress = (!idView.empty() && total > 0 && !isLog);

if (m_verbose || !m_isConsole)
{
wprintf(L"%hs", status);
// Skip pull progress updates when output is redirected, show only major steps
if (!isPullProgress)
{
wprintf(L"%hs", status);
}
return S_OK;
}

// Match the specific "log" sentinel sent by WSLCSession::BuildImage rather than
// accepting any non-empty id, so future or unrelated id usage defaults to permanent.
const bool isLog = (id != nullptr && std::string_view{id} == "log");

if (!isLog)
// Pull/download progress: update the per-entry map so Redraw can show each entry
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we are handling this correctly when m_isConsole is false. We should probably skip the whole callback stuff when non in an interactive terminal.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would break AIs who are scraping wslc.exe build output. Right now we output something like this:

PS C:\cDev\copilot\build-test> cat .\dockerfilesimple.log
Building image from directory: .

[1/2] FROM docker.io/library/ubuntu:24.04@sha256:c4a8d5503dfb2a3eb8ab5f807da5bc69a85730fb49b5cfca2330194ebcc41c7b
[2/2] RUN echo 123
exporting to image
  | exporting layers
  | writing image sha256:780c43a5ab95735b080898e63db96c3feebdd05591a0a934bc3252fe70201004
  | naming to docker.io/library/test
PS C:\cDev\copilot\build-test>

I think this is pretty reasonable and gives the right info on all the steps.

// on a single line that updates in place.
Comment thread
craigloewen-msft marked this conversation as resolved.
if (isPullProgress)
{
// Permanent line: collapse the scrolling window then print directly.
CollapseWindow();
WriteTerminal(MultiByteToWide(status));
m_pullLines[id] = status;

auto now = std::chrono::steady_clock::now();
if (now - m_lastRedraw >= c_redrawInterval)
{
Redraw();
m_lastRedraw = now;
}

return S_OK;
}
Comment thread
craigloewen-msft marked this conversation as resolved.

// Log line: add to the scrolling window.
for (const char* p = status; *p != '\0'; ++p)
if (isLog)
{
if (*p == '\n')
// Log line: add to the scrolling window.
for (const char* p = status; *p != '\0'; ++p)
{
// Store with the trailing newline so the byte count matches what is replayed.
// Cap retained log output to avoid unbounded growth on very long builds.
m_allLines.push_back(m_pendingLine + '\n');
m_allLinesBytes += m_allLines.back().size();
while (m_allLinesBytes > c_maxAllLinesBytes && !m_allLines.empty())
if (*p == '\n')
{
m_allLinesBytes -= m_allLines.front().size();
m_allLines.pop_front();
}
// Store with the trailing newline so the byte count matches what is replayed.
// Cap retained log output to avoid unbounded growth on very long builds.
m_allLines.push_back(m_pendingLine + '\n');
m_allLinesBytes += m_allLines.back().size();
while (m_allLinesBytes > c_maxAllLinesBytes && !m_allLines.empty())
{
m_allLinesBytes -= m_allLines.front().size();
m_allLines.pop_front();
}

m_lines.push_back(std::move(m_pendingLine));
m_pendingLine.clear();
if (m_lines.size() > c_maxDisplayLines)
{
m_lines.pop_front();
m_lines.push_back(std::move(m_pendingLine));
m_pendingLine.clear();
if (m_lines.size() > c_maxDisplayLines)
{
m_lines.pop_front();
}
}
}
else if (*p == '\r')
{
// \r\n is a line ending; standalone \r overwrites the current line.
if (*(p + 1) != '\n')
else if (*p == '\r')
{
// Flush a throttled redraw before clearing so \r-based progress
// updates are visible even when batched in a single OnProgress call.
auto now = std::chrono::steady_clock::now();
if (!m_pendingLine.empty() && now - m_lastRedraw >= c_redrawInterval)
// \r\n is a line ending; standalone \r overwrites the current line.
if (*(p + 1) != '\n')
{
Redraw();
m_lastRedraw = now;
// Flush a throttled redraw before clearing so \r-based progress
// updates are visible even when batched in a single OnProgress call.
auto now = std::chrono::steady_clock::now();
if (!m_pendingLine.empty() && now - m_lastRedraw >= c_redrawInterval)
{
Redraw();
m_lastRedraw = now;
}
m_pendingLine.clear();
}
m_pendingLine.clear();
}
else
{
m_pendingLine += *p;
}
}
else

// Throttle redraws to avoid blocking the server's IO loop with console writes
// during rapid output. Lines accumulate in the deque immediately; the display
// catches up at ~20fps.
auto now = std::chrono::steady_clock::now();
if (now - m_lastRedraw >= c_redrawInterval)
{
m_pendingLine += *p;
Redraw();
m_lastRedraw = now;
}
}

// Throttle redraws to avoid blocking the server's IO loop with console writes
// during rapid output. Lines accumulate in the deque immediately; the display
// catches up at ~20fps.
auto now = std::chrono::steady_clock::now();
if (now - m_lastRedraw >= c_redrawInterval)
{
Redraw();
m_lastRedraw = now;
return S_OK;
}

// Else is a build step
CollapseWindow();
auto wide = MultiByteToWide(status);
const auto bodyLength = wide.find_last_not_of(L"\r\n") + 1;
const auto newlines = wide.substr(bodyLength);
wide.resize(bodyLength);

WriteTerminal(std::format(L"\033[92m{}\033[0m{}", wide, newlines));
return S_OK;
}
CATCH_RETURN();
Expand All @@ -167,14 +193,16 @@ void BuildImageCallback::Redraw()
// to std::wstring::resize).
const SHORT consoleWidth = std::max<SHORT>(0, info.srWindow.Right - info.srWindow.Left);

// Determine how many completed lines to show, leaving room for the pending line.
// Determine how many completed lines to show, leaving room for the pending line and pull progress.
const bool showPending = !m_pendingLine.empty();
const SHORT pullCount = static_cast<SHORT>(m_pullLines.size());
SHORT completedCount = static_cast<SHORT>(m_lines.size());
if (showPending && completedCount >= c_maxDisplayLines)
const SHORT reservedLines = (showPending ? 1 : 0) + pullCount;
if (completedCount + reservedLines > c_maxDisplayLines)
{
completedCount = c_maxDisplayLines - 1;
completedCount = std::max<SHORT>(0, c_maxDisplayLines - reservedLines);
}
const SHORT displayCount = completedCount + (showPending ? 1 : 0);
const SHORT displayCount = completedCount + reservedLines;

// Build the entire frame in one buffer to minimize console writes. Hide the cursor
// during the redraw so the user doesn't see it bouncing through the cursor movement,
Expand Down Expand Up @@ -217,6 +245,12 @@ void BuildImageCallback::Redraw()
appendLine(m_pendingLine);
}

// Render per-entry pull progress (each entry updates in place via the map).
for (const auto& [key, line] : m_pullLines)
Comment thread
craigloewen-msft marked this conversation as resolved.
{
appendLine(line);
}

buffer += L"\033[22m\033[?25h";

WriteTerminal(buffer);
Expand Down
3 changes: 3 additions & 0 deletions src/windows/wslc/services/BuildImageCallback.h
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Module Name:
#include "ChangeTerminalMode.h"
#include "SessionService.h"
#include <deque>
#include <map>

namespace wsl::windows::wslc::services {
class DECLSPEC_UUID("3EDD5DBF-CA6C-4CF7-923A-AD94B6A732E5") BuildImageCallback
Expand Down Expand Up @@ -54,6 +55,8 @@ class DECLSPEC_UUID("3EDD5DBF-CA6C-4CF7-923A-AD94B6A732E5") BuildImageCallback
std::string m_pendingLine;
SHORT m_displayedLines = 0;
std::chrono::steady_clock::time_point m_lastRedraw{};
// Per-entry pull progress lines, keyed by entry id. Updated in place by Redraw. std::map so order is consistent.
std::map<std::string, std::string> m_pullLines;
// Captured at construction so the destructor can detect destruction during exception unwinding.
int m_uncaughtExceptions = std::uncaught_exceptions();
};
Expand Down
47 changes: 41 additions & 6 deletions src/windows/wslc/services/ImageProgressCallback.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -84,23 +84,58 @@ std::wstring ImageProgressCallback::GenerateStatusLine(LPCSTR status, LPCSTR id,
std::wstring line;
if (total != 0)
{
line = std::format(L"{} '{}': {}%", status, id, current * 100 / total);
constexpr int c_progressBarWidth = 30;

int filled = 0;
if (current >= total)
{
filled = c_progressBarWidth;
}
else
{
auto ratio = static_cast<long double>(current) / static_cast<long double>(total);
filled = static_cast<int>(ratio * c_progressBarWidth);
}

filled = std::clamp(filled, 0, c_progressBarWidth);

std::wstring bar;
bar.reserve(c_progressBarWidth);
bar.append(filled, L'=');
bar.append(L">");
bar.resize(c_progressBarWidth, L' ');

line = std::format(
L"{}: {} [{}] {}/{}", id, status, bar, wsl::shared::string::FormatBytes(current), wsl::shared::string::FormatBytes(total));
}
else if (current != 0)
{
line = std::format(L"{} '{}': {}s", status, id, current);
line = std::format(L"{}: {} {}s", id, status, current);
}
else
{
line = std::format(L"{} '{}'", status, id);
line = std::format(L"{}: {}", id, status);
}

// Erase any previously written char on that line.
while (line.size() < info.dwSize.X)
// Use the visible window width (not the buffer width) to prevent wrapping.
const auto visibleWidth = std::max<SHORT>(0, info.srWindow.Right - info.srWindow.Left + 1);

// Truncate to console width to prevent wrapping that would break cursor repositioning.
if (line.size() > static_cast<size_t>(visibleWidth))
Comment on lines +120 to +124
{
line += L' ';
line.resize(visibleWidth);

// Avoid splitting a surrogate pair — if the last code unit is a high surrogate,
// drop it so we don't emit an invalid UTF-16 sequence.
if (!line.empty() && IS_HIGH_SURROGATE(line.back()))
{
line.pop_back();
}
}
Comment thread
craigloewen-msft marked this conversation as resolved.

// Erase any previously written char on that line.
line.resize(visibleWdith, L' ');

return line;
}
} // namespace wsl::windows::wslc::services
21 changes: 17 additions & 4 deletions src/windows/wslcsession/WSLCSession.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -827,10 +827,10 @@ try
return " [" + name + "] ";
};

auto reportProgress = [&](const std::string& message, const char* id = "") {
auto reportProgress = [&](const std::string& message, const char* id = "", ULONGLONG current = 0, ULONGLONG total = 0) {
if (ProgressCallback != nullptr)
{
THROW_IF_FAILED(ProgressCallback->OnProgress(message.c_str(), id, 0, 0));
THROW_IF_FAILED(ProgressCallback->OnProgress(message.c_str(), id, current, total));
}
};

Expand Down Expand Up @@ -924,8 +924,21 @@ try

for (const auto& entry : status.statuses)
{
if (auto it = digestToStageName.find(entry.vertex);
it != digestToStageName.end() && !entry.id.empty() && reportedSteps.insert(entry.id).second)
auto it = digestToStageName.find(entry.vertex);
if (it == digestToStageName.end() || entry.id.empty())
{
continue;
}

if (entry.total > 0)
{
auto currentBytes = static_cast<ULONGLONG>(std::max<int64_t>(entry.current, 0));
auto totalBytes = static_cast<ULONGLONG>(std::max<int64_t>(entry.total, 0));
auto current = wsl::shared::string::FormatBytes(currentBytes);
auto total = wsl::shared::string::FormatBytes(totalBytes);
reportProgress(std::format("{}{} {} / {}", logPrefix(it->second), entry.id, current, total), entry.id.c_str(), currentBytes, totalBytes);
}
else if (reportedSteps.insert(entry.id).second)
{
flushLine();
reportProgress(logPrefix(it->second) + entry.id + "\n");
Expand Down
Loading