Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/add acceptance ratebycount for issue 113 {Show the Acceptance Rate (by count) and Acceptance Rate (by lines) #113} #115

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,44 +16,44 @@ https://github.com/github-copilot-resources/copilot-metrics-viewer/assets/332930

## Key Metrics
Here are the key metrics visualized in these charts:
1. **Acceptance Rate:** This metric represents the ratio of accepted lines to the total lines suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.
1. **Acceptance Rate:** This metric represents the ratio of accepted numbers to the total numbers suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.
<p align="center">
<img width="800" alt="image" src="https://github.com/martedesco/copilot-metrics-viewer/assets/3329307/875a5f5f-5d8a-44bd-a4e9-0f663f2b2628">
<img width="800" alt="image" src="./images/Acceptance_rate_bycount.png">
</p>

2. **Total Suggestions** This chart illustrates the total number of code suggestions made by GitHub Copilot. It offers a view of the tool's activity and its engagement with users over time.

3. **Total Acceptances:** This visualization focuses on the total number of suggestions accepted by users.

<p align="center">
<img width="800" alt="image" src="https://github.com/martedesco/copilot-metrics-viewer/assets/3329307/b84220ae-fbdc-4503-b50b-4689362bf364">
<img width="800" alt="image" src="./images/Total_suggestions_count.png">
</p>

4. **Total Lines Suggested:** Showcases the total number of lines of code suggested by GitHub Copilot. This gives an idea of the volume of code generation and assistance provided.

5. **Total Lines Accepted:** As the name says, the total lines of code accepted by users (full acceptances) offering insights into how much of the suggested code is actually being utilized incorporated to the codebase.

<p align="center">
<img width="800" alt="image" src="https://github.com/martedesco/copilot-metrics-viewer/assets/3329307/788c9b33-8e63-43a5-9ab9-98d8938dd9d9">
<img width="800" alt="image" src="./images/Total Lines.png">
</p>

6. **Total Active Users:** Represents the number of active users engaging with GitHub Copilot. This helps in understanding the user base growth and adoption rate.

<p align="center">
<img width="800" alt="image" src="https://github.com/martedesco/copilot-metrics-viewer/assets/3329307/bd92918f-3a11-492b-8490-877aaa768ca3">
<img width="800" alt="image" src="./images/Total_Active_users.png">
</p>

## Languages Breakdown Analysis

Pie charts with the top 5 languages by accepted prompts and acceptance rate are displayed at the top.
Pie charts with the top 5 languages by accepted prompts and acceptance rate (by count/by lines) are displayed at the top.
<p align="center">
<img width="800" alt="image" src="https://github.com/github-copilot-resources/copilot-metrics-viewer/assets/3329307/8ab0488a-89e6-486d-aa61-df3d178cd57c">
<img width="800" alt="image" src="./images/Language_breakdown.png">
</p>

The language breakdown analysis tab also displays a table showing the Accepted Prompts, Accepted Lines of Code, and Acceptance Rate (%) for each language over the past 28 days. The entries are sorted by the number of _accepted lines of code descending_.

<p align="center">
<img width="800" alt="image" src="https://github.com/github-copilot-resources/copilot-metrics-viewer/assets/3329307/38a4ff57-4974-4f60-a154-91db17b03678">
<img width="800" alt="image" src="./images/Language_breakdown_list.png">
</p>

## Copilot Chat Metrics
Expand Down
Binary file added images/Acceptance_rate_by_lines.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Acceptance_rate_bycount.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Language_breakdown.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Language_breakdown_list.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/MainMetrics.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Total Lines.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Total_Active_users.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Total_suggestions_count.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
91 changes: 63 additions & 28 deletions src/components/BreakdownComponent.vue
Original file line number Diff line number Diff line change
Expand Up @@ -19,25 +19,37 @@
<v-main class="p-1" style="min-height: 300px;">
<v-container style="min-height: 300px;" class="px-4 elevation-2">
<v-row>
<v-col cols="6">
<v-col cols="4">
<v-card>
<v-card-item class="d-flex justify-center align-center">
<div class="spacing-25"></div>
<div class="text-h6 mb-1">Top 5 {{ breakdownDisplayNamePlural }} by accepted prompts</div>
<div class="text-h6 mb-1">Top 5 {{ breakdownDisplayNamePlural }} by accepted suggestions(prompts)</div>
<div style="width: 300px; height: 300px;">
<Pie :data="breakdownsChartDataTop5AcceptedPrompts" :options="chartOptions" />
</div>
</v-card-item>
</v-card>
</v-col>

<v-col cols="6">
<v-col cols="4">
<v-card>
<v-card-item class="d-flex justify-center align-center">
<div class="spacing-25"></div>
<div class="text-h6 mb-1">Top 5 {{ breakdownDisplayNamePlural }} by acceptance rate</div>
<div class="text-h6 mb-1">Acceptance Rate (by count) for Top 5 {{ breakdownDisplayNamePlural }}</div>
<div style="width: 300px; height: 300px;">
<Pie :data="breakdownsChartDataTop5AcceptanceRate" :options="chartOptions" />
<Pie :data="breakdownsChartDataTop5AcceptedPromptsByCounts" :options="chartOptions" />
</div>
</v-card-item>
</v-card>
</v-col>

<v-col cols="4">
<v-card>
<v-card-item class="d-flex justify-center align-center">
<div class="spacing-25"></div>
<div class="text-h6 mb-1">Acceptance Rate (by code lines) for Top 5 {{ breakdownDisplayNamePlural }}</div>
<div style="width: 300px; height: 300px;">
<Pie :data="breakdownsChartDataTop5AcceptedPromptsByLines" :options="chartOptions" />
</div>
</v-card-item>
</v-card>
Expand All @@ -53,8 +65,11 @@
<tr>
<td>{{ item.name }}</td>
<td>{{ item.acceptedPrompts }}</td>
<td>{{ item.suggestedPrompts }}</td>
<td>{{ item.acceptedLinesOfCode }}</td>
<td v-if="item.acceptanceRate !== undefined">{{ item.acceptanceRate.toFixed(2) }}%</td>
<td>{{ item.suggestedLinesOfCode }}</td>
<td v-if="item.acceptanceRateByCount !== undefined">{{ item.acceptanceRateByCount.toFixed(2) }}%</td>
<td v-if="item.acceptanceRateByLines !== undefined">{{ item.acceptanceRateByLines.toFixed(2) }}%</td>
</tr>
</template>
</v-data-table>
Expand All @@ -74,9 +89,9 @@ import {
ArcElement,
CategoryScale,
LinearScale,
BarElement,
PointElement,
LineElement,
BarElement,
Title,
Tooltip,
Legend
Expand Down Expand Up @@ -120,8 +135,11 @@ export default defineComponent({
return [
{ title: `${this.breakdownDisplayName} Name`, key: 'name' },
{ title: 'Accepted Prompts', key: 'acceptedPrompts' },
{ title: 'Suggested Prompts', key: 'suggestedPrompts' },
{ title: 'Accepted Lines of Code', key: 'acceptedLinesOfCode' },
{ title: 'Acceptance Rate (%)', key: 'acceptanceRate' },
{ title: 'Suggested Lines of Code', key: 'suggestedLinesOfCode' },
{ title: 'Acceptance Rate by Count (%)', key: 'acceptanceRateByCount' },
{ title: 'Acceptance Rate by Lines (%)', key: 'acceptanceRateByLines' },
];
},
},
Expand All @@ -139,8 +157,11 @@ export default defineComponent({
//Top 5 by accepted prompts
let breakdownsChartDataTop5AcceptedPrompts = ref<{ labels: string[]; datasets: any[] }>({ labels: [], datasets: [] });

//Top 5 by acceptance rate
let breakdownsChartDataTop5AcceptanceRate = ref<{ labels: string[]; datasets: any[] }>({ labels: [], datasets: [] });
//Acceptance Rate by lines for top 5 by accepted prompts
let breakdownsChartDataTop5AcceptedPromptsByLines = ref<{ labels: string[]; datasets: any[] }>({ labels: [], datasets: [] });

//Acceptance Rate by counts for top 5 by accepted prompts
let breakdownsChartDataTop5AcceptedPromptsByCounts = ref<{ labels: string[]; datasets: any[] }>({ labels: [], datasets: [] });

const chartOptions = {
responsive: true,
Expand All @@ -157,6 +178,7 @@ export default defineComponent({
// Process the breakdown separately
data.forEach((m: Metrics) => m.breakdown.forEach(breakdownData =>
{
//console.log('Processing breakdown data:', breakdownData);
const breakdownName = breakdownData[props.breakdownKey as keyof typeof breakdownData] as string;
let breakdown = breakdownList.value.find(b => b.name === breakdownName);

Expand All @@ -165,47 +187,60 @@ export default defineComponent({
breakdown = new Breakdown({
name: breakdownName,
acceptedPrompts: breakdownData.acceptances_count,
suggestedPrompts: breakdownData.suggestions_count,
suggestedLinesOfCode: breakdownData.lines_suggested,
acceptedLinesOfCode: breakdownData.lines_accepted,
});
breakdownList.value.push(breakdown);
} else {
// Update the existing breakdown object
breakdown.acceptedPrompts += breakdownData.acceptances_count;
breakdown.suggestedPrompts += breakdownData.suggestions_count;
breakdown.suggestedLinesOfCode += breakdownData.lines_suggested;
breakdown.acceptedLinesOfCode += breakdownData.lines_accepted;
}
// Recalculate the acceptance rate
breakdown.acceptanceRate = breakdown.suggestedLinesOfCode !== 0 ? (breakdown.acceptedLinesOfCode / breakdown.suggestedLinesOfCode) * 100 : 0;
}));
// Recalculate the acceptance rates
breakdown.acceptanceRateByCount = breakdown.suggestedPrompts !== 0 ? (breakdown.acceptedPrompts / breakdown.suggestedPrompts) * 100 : 0;
breakdown.acceptanceRateByLines = breakdown.suggestedLinesOfCode !== 0 ? (breakdown.acceptedLinesOfCode / breakdown.suggestedLinesOfCode) * 100 : 0;

//Sort breakdowns map by acceptance rate
breakdownList.value.sort((a, b) => b.acceptanceRate - a.acceptanceRate);
// Log each breakdown for debugging
// console.log('Breakdown:', breakdown);
}));

// Get the top 5 breakdowns by acceptance rate
const top5BreakdownsAcceptanceRate = breakdownList.value.slice(0, 5);
// for test, it seems there is an issue in data, so need to get the data in console
console.log('Breakdown List:', JSON.stringify(breakdownList.value, null, 2));

//Sort breakdowns map by accepted prompts
breakdownList.value.sort((a, b) => b.acceptedPrompts - a.acceptedPrompts);

breakdownsChartDataTop5AcceptanceRate.value = {
labels: top5BreakdownsAcceptanceRate.map(breakdown => breakdown.name),
// Get the top 5 breakdowns by accepted prompts
const top5BreakdownsAcceptedPrompts = breakdownList.value.slice(0, 5);

breakdownsChartDataTop5AcceptedPrompts.value = {
labels: top5BreakdownsAcceptedPrompts.map(breakdown => breakdown.name),
datasets: [
{
data: top5BreakdownsAcceptanceRate.map(breakdown => breakdown.acceptanceRate.toFixed(2)),
data: top5BreakdownsAcceptedPrompts.map(breakdown => breakdown.acceptedPrompts),
backgroundColor: pieChartColors.value,
},
],
};

//Sort breakdowns map by accepted prompts
breakdownList.value.sort((a, b) => b.acceptedPrompts - a.acceptedPrompts);
breakdownsChartDataTop5AcceptedPromptsByLines.value = {
labels: top5BreakdownsAcceptedPrompts.map(breakdown => breakdown.name),
datasets: [
{
data: top5BreakdownsAcceptedPrompts.map(breakdown => breakdown.acceptanceRateByLines.toFixed(2)),
backgroundColor: pieChartColors.value,
},
],
};

// Get the top 5 breakdowns by accepted prompts
const top5BreakdownsAcceptedPrompts = breakdownList.value.slice(0, 5);

breakdownsChartDataTop5AcceptedPrompts.value = {
breakdownsChartDataTop5AcceptedPromptsByCounts.value = {
labels: top5BreakdownsAcceptedPrompts.map(breakdown => breakdown.name),
datasets: [
{
data: top5BreakdownsAcceptedPrompts.map(breakdown => breakdown.acceptedPrompts),
data: top5BreakdownsAcceptedPrompts.map(breakdown => breakdown.acceptanceRateByCount.toFixed(2)),
backgroundColor: pieChartColors.value,
},
],
Expand All @@ -214,7 +249,7 @@ export default defineComponent({
numberOfBreakdowns.value = breakdownList.value.length;

return { chartOptions, breakdownList, numberOfBreakdowns,
breakdownsChartData, breakdownsChartDataTop5AcceptedPrompts, breakdownsChartDataTop5AcceptanceRate };
breakdownsChartData, breakdownsChartDataTop5AcceptedPrompts, breakdownsChartDataTop5AcceptedPromptsByLines, breakdownsChartDataTop5AcceptedPromptsByCounts };
},


Expand Down
32 changes: 20 additions & 12 deletions src/components/MainComponent.vue
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,11 @@ export default defineComponent({
const apiError = ref<string | undefined>(undefined);
const signInRequired = ref(false);

function processError(error: any) {
/**
* Handles API errors by setting appropriate error messages.
* @param {any} error - The error object returned from the API call.
*/
function processError(error: any) {
console.log(error);
// Check the status code of the error response
if (error.response && error.response.status) {
Expand All @@ -153,20 +157,23 @@ export default defineComponent({
if(config.github.team && config.github.team.trim() !== '') {
getTeamMetricsApi().then(data => {
metrics.value = data;
//console.log("Metrics data in getTeamMetricsApi: ", metrics.value);

// Set metricsReady to true after the call completes.
metricsReady.value = true;
}).catch(processError);
}

if (metricsReady.value === false) {
getMetricsApi().then(data => {
metrics.value = data;

// Set metricsReady to true after the call completes.
metricsReady.value = true;

}).catch(processError);
else {
if (metricsReady.value === false) {
getMetricsApi().then(data => {
metrics.value = data;
//console.log("Metrics data in getMetricsApi: ", metrics.value);

// Set metricsReady to true after the call completes.
metricsReady.value = true;

}).catch(processError);
}
}

getSeatsApi().then(data => {
Expand All @@ -179,7 +186,8 @@ export default defineComponent({

return { metricsReady, metrics, seatsReady, seats, apiError, signInRequired };
}
})
}
)
</script>

<style scoped>
Expand Down Expand Up @@ -217,4 +225,4 @@ export default defineComponent({
.github-login-button v-icon {
margin-right: 8px;
}
</style>
</style>
Loading