Skip to content

Commit 7f2ca04

Browse files
pkufoolcsukuangfjpzelaskojtrmalluomingshuang
authored
update v2.0-pre (#953)
* Update doc URL. (#821) * Support indexing 2-axes RaggedTensor, Support slicing for RaggedTensor (#825) * Support index 2-axes RaggedTensor, Support slicing for RaggedTensor * Fix compiling errors * Fix unit test * Change RaggedTensor.data to RaggedTensor.values * Fix style * Add docs * Run nightly-cpu when pushing code to nightly-cpu branch * Prune with max_arcs in IntersectDense (#820) * Add checking for array constructor * Prune with max arcs * Minor fix * Fix typo * Fix review comments * Fix typo * Release v1.8 * Create a ragged tensor from a regular tensor. (#827) * Create a ragged tensor from a regular tensor. * Add tests for creating ragged tensors from regular tensors. * Add more tests. * Print ragged tensors in a way like what PyTorch is doing. * Fix test cases. * Trigger GitHub actions manually. (#829) * Run GitHub actions on merging. (#830) * Support printing ragged tensors in a more compact way. (#831) * Support printing ragged tensors in a more compact way. * Disable support for torch 1.3.1 * Fix test failures. * Add levenshtein alignment (#828) * Add levenshtein graph * Contruct k2.RaggedTensor in python part * Fix review comments, return aux_labels in ctc_graph * Fix tests * Fix bug of accessing symbols * Fix bug of accessing symbols * Change argument name, add levenshtein_distance interface * Fix test error, add tests for levenshtein_distance * Fix review comments and add unit test for c++ side * update the interface of levenshtein alignment * Fix review comments * Release v1.9 * Support a[b[i]] where both a and b are ragged tensors. (#833) * Display import error solution message on MacOS (#837) * Fix installation doc. (#841) * Fix installation doc. Remove Windows support. Will fix it later. * Fix style issues. * fix typos in the install instructions (#844) * make cmake adhere to the modernized way of finding packages outside default dirs (#845) * import torch first in the smoke tests to preven SEGFAULT (#846) * Add doc about how to install a CPU version of k2. (#850) * Add doc about how to install a CPU version of k2. * Remove property setter of Fsa.labels * Update Ubuntu version in GitHub CI since 16.04 reaches end-of-life. * Support PyTorch 1.10. (#851) * Fix test cases for k2.union() (#853) * Fix out-of-boundary access (read). (#859) * Update all the example codes in the docs (#861) * Update all the example codes in the docs I have run all the modified codes with the newest version k2. * do some changes * Fix compilation errors with CUB 1.15. (#865) * Update README. (#873) * Update README. * Fix typos. * Fix ctc graph (make aux_labels of final arcs -1) (#877) * Fix LICENSE location to k2 folder (#880) * Release v1.11. (#881) It contains bugfixes. * Update documentation for hash.h (#887) * Update documentation for hash.h * Typo fix * Wrap MonotonicLowerBound (#883) * Wrap MonotonicLowerBound * Add unit tests * Support int64; update documents * Remove extra commas after 'TOPSORTED' properity and fix RaggedTensor constructer parameter 'byte_offset' out-of-range bug. (#892) Co-authored-by: gzchenduisheng <gzchenduisheng@corp.netease.com> * Fix small typos (#896) * Fix k2.ragged.create_ragged_shape2 (#901) Before the fix, we have to specify both `row_splits` and `row_ids` while calling `k2.create_ragged_shape2` even if one of them is `None`. After this fix, we only need to specify one of them. * Add rnnt loss (#891) * Add cpp code of mutual information * mutual information working * Add rnnt loss * Add pruned rnnt loss * Minor Fixes * Minor fixes & fix code style * Fix cpp style * Fix code style * Fix s_begin values in padding positions * Fix bugs related to boundary; Fix s_begin padding value; Add more tests * Minor fixes * Fix comments * Add boundary to pruned loss tests * Use more efficient way to fix boundaries (#906) * Release v1.12 (#907) * Change the sign of the rnnt_loss and add reduction argument (#911) * Add right boundary constrains for s_begin * Minor fixes to the interface of rnnt_loss to make it return positive value * Fix comments * Release a new version * Minor fixes * Minor fixes to the docs * Fix building doc. (#908) * Fix building doc. * Minor fixes. * Minor fixes. * Fix building doc (#912) * Fix building doc * Fix flake8 * Support torch 1.10.x (#914) * Support torch 1.10.x * Fix installing PyTorch. * Update INSTALL.rst (#915) * Update INSTALL.rst Setting a few additional env variables to enable compilation from source *with CUDA GPU computation support enabled* * Fix torch/cuda/python versions in the doc. (#918) * Fix torch/cuda/python versions in the doc. * Minor fixes. * Fix building for CUDA 11.6 (#917) * Fix building for CUDA 11.6 * Minor fixes. * Implement Unstack (#920) * Implement unstack * Remove code does not relate to this PR * Remove for loop on output dim; add Unstack ragged * Add more docs * Fix comments * Fix docs & unit tests * SubsetRagged & PruneRagged (#919) * Extend interface of SubsampleRagged. * Add interface for pruning ragged tensor. * Draft of new RNN-T decoding method * Implements SubsampleRaggedShape * Implements PruneRagged * Rename subsample-> subset * Minor fixes * Fix comments Co-authored-by: Daniel Povey <dpovey@gmail.com> * Add Hash64 (#895) * Add hash64 * Fix tests * Resize hash64 * Fix comments * fix typo * Modified rnnt (#902) * Add modified mutual_information_recursion * Add modified rnnt loss * Using more efficient way to fix boundaries * Fix modified pruned rnnt loss * Fix the s_begin constrains of pruned loss for modified version transducer * Fix Stack (#925) * return the correct layer * unskip the test * Fix 'TypeError' of rnnt_loss_pruned function. (#924) * Fix 'TypeError' of rnnt_loss_simple function. Fix 'TypeError' exception when calling rnnt_loss_simple(..., return_grad=False) at validation steps. * Fix 'MutualInformationRecursionFunction.forward()' return type check error for pytorch < 1.10.x * Modify return type. * Add documents about class MutualInformationRecursionFunction. * Formated code style. * Fix rnnt_loss_smoothed return type. Co-authored-by: gzchenduisheng <gzchenduisheng@corp.netease.com> * Support torch 1.11.0 and CUDA 11.5 (#931) * Support torch 1.11.0 and CUDA 11.5 * Implement Rnnt decoding (#926) * first working draft of rnnt decoding * FormatOutput works... * Different num frames for FormatOutput works * Update docs * Fix comments, break advance into several stages, add more docs * Add python wrapper * Add more docs * Minor fixes * Fix comments * fix building docs (#933) * Release v1.14 * Remove unused DiscountedCumSum. (#936) * Fix compiler warnings. (#937) * Fix compiler warnings. * Minor fixes for RNN-T decoding. (#938) * Minor fixes for RNN-T decoding. * Removes arcs with label 0 from the TrivialGraph. (#939) * Implement linear_fsa_with_self_loops. (#940) * Implement linear_fsa_with_self_loops. * Fix the pruning with max-states (#941) * Rnnt allow different encoder/decoder dims (#945) * Allow different encoder and decoder dim in rnnt_pruning * Bug fixes * Supporting building k2 on Windows (#946) * Fix nightly windows CPU build (#948) * Fix nightly building k2 for windows. * Run nightly build only if there are new commits. * Check the versions of PyTorch and CUDA at the import time. (#949) * Check the versions of PyTorch and CUDA at the import time. * More straightforward message when CUDA support is missing (#950) * Implement ArrayOfRagged (#927) * Implement ArrayOfRagged * Fix issues and pass tests * fix style * change few statements of functions and move the definiation of template Array1OfRagged to header file * add offsets test code * Fix precision (#951) * Fix precision * Using different pow version for windows and *nix * Use int64_t pow * Minor fixes Co-authored-by: Fangjun Kuang <csukuangfj@gmail.com> Co-authored-by: Piotr Żelasko <petezor@gmail.com> Co-authored-by: Jan "yenda" Trmal <jtrmal@gmail.com> Co-authored-by: Mingshuang Luo <37799481+luomingshuang@users.noreply.github.com> Co-authored-by: Ludwig Kürzinger <lumaku@users.noreply.github.com> Co-authored-by: Daniel Povey <dpovey@gmail.com> Co-authored-by: drawfish <duisheng.chen@gmail.com> Co-authored-by: gzchenduisheng <gzchenduisheng@corp.netease.com> Co-authored-by: alexei-v-ivanov <alexei_v_ivanov@ieee.org> Co-authored-by: Wang, Guanbo <wgb14@outlook.com> Co-authored-by: Nickolay V. Shmyrev <nshmyrev@gmail.com> Co-authored-by: LvHang <hanglyu1991@gmail.com>
1 parent 1d30b9a commit 7f2ca04

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+1706
-1207
lines changed

‎.flake8

+1
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ exclude =
1414
get_version.py
1515
build,
1616
k2/python/host,
17+
k2/python/k2/__init__.py,
1718
k2/python/k2/ctc_loss.py,
1819
docs
1920

‎.github/workflows/build-conda-cpu-macos.yml

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Copyright 2021 Xiaomi Corp. (author: Fangjun Kuang)
2+
3+
# See ../../LICENSE for clarification regarding multiple authors
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# refer to https://github.com/actions/starter-workflows/pull/47/files
18+
19+
20+
# Note, we have to set
21+
#
22+
# export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH
23+
#
24+
# before running `python3 -m k2.version`
25+
#
26+
# See https://github.com/openPMD/openPMD-api/issues/593#issuecomment-552690470
27+
28+
29+
name: build_conda_cpu_macos
30+
31+
on:
32+
push:
33+
tags:
34+
- '*'
35+
36+
env:
37+
K2_BUILD_TYPE: Release
38+
39+
jobs:
40+
generate_build_matrix:
41+
# see https://github.com/pytorch/pytorch/pull/50633
42+
runs-on: ubuntu-latest
43+
outputs:
44+
matrix: ${{ steps.set-matrix.outputs.matrix }}
45+
steps:
46+
- uses: actions/checkout@v2
47+
with:
48+
fetch-depth: 0
49+
- name: Generating build matrix
50+
id: set-matrix
51+
run: |
52+
# outputting for debugging purposes
53+
python scripts/github_actions/generate_build_matrix.py
54+
MATRIX=$(python scripts/github_actions/generate_build_matrix.py)
55+
echo "::set-output name=matrix::${MATRIX}"
56+
57+
build_conda_cpu_macos:
58+
needs: generate_build_matrix
59+
runs-on: macos-10.15
60+
strategy:
61+
fail-fast: false
62+
matrix:
63+
${{ fromJson(needs.generate_build_matrix.outputs.matrix) }}
64+
65+
steps:
66+
# refer to https://github.com/actions/checkout
67+
- uses: actions/checkout@v2
68+
with:
69+
fetch-depth: 0
70+
71+
- uses: conda-incubator/setup-miniconda@v2
72+
with:
73+
auto-update-conda: true
74+
python-version: ${{ matrix.python-version }}
75+
activate-environment: k2
76+
77+
- name: Display Python version
78+
shell: bash -l {0}
79+
run: |
80+
python3 -c "import sys; print(sys.version)"
81+
which python3
82+
83+
- name: Install conda dependencies
84+
shell: bash -l {0}
85+
run: |
86+
conda install -y -q anaconda-client
87+
conda install -y -q conda-build
88+
conda install -y -q -c pytorch pytorch=${{ matrix.torch }} cpuonly
89+
90+
- name: Display conda info
91+
shell: bash -l {0}
92+
run: |
93+
which conda
94+
conda env list
95+
conda info
96+
97+
- name: Build k2
98+
shell: bash -l {0}
99+
env:
100+
K2_PYTHON_VERSION: ${{ matrix.python-version}}
101+
K2_TORCH_VERSION: ${{ matrix.torch }}
102+
K2_CONDA_TOKEN: ${{ secrets.K2_CONDA_TOKEN}}
103+
K2_IS_GITHUB_ACTIONS: 1
104+
K2_IS_FOR_CONDA: 1
105+
run: |
106+
export K2_BUILD_TYPE=$K2_BUILD_TYPE
107+
./scripts/build_conda_cpu.sh
108+
109+
- name: Display generated files
110+
run: |
111+
ls -lh /usr/local/miniconda/envs/k2/conda-bld/osx-64
112+
113+
- name: Upload generated files
114+
uses: actions/upload-artifact@v2
115+
with:
116+
name: torch-${{ matrix.torch }}-python-${{ matrix.python-version }}-${{ matrix.os }}
117+
path: /usr/local/miniconda/envs/k2/conda-bld/osx-64/*.tar.bz2

‎.github/workflows/build_conda_cpu.yml ‎.github/workflows/build-conda-cpu-ubuntu.yml

+24-65
Original file line numberDiff line numberDiff line change
@@ -26,68 +26,41 @@
2626
# See https://github.com/openPMD/openPMD-api/issues/593#issuecomment-552690470
2727

2828

29-
name: build_conda_cpu
29+
name: build_conda_cpu_ubuntu
3030

3131
on:
3232
push:
33-
branches:
34-
- conda-cpu
33+
tags:
34+
- '*'
3535

3636
env:
3737
K2_BUILD_TYPE: Release
3838

3939
jobs:
40-
build_conda_cpu:
41-
runs-on: ${{ matrix.os }}
40+
generate_build_matrix:
41+
# see https://github.com/pytorch/pytorch/pull/50633
42+
runs-on: ubuntu-latest
43+
outputs:
44+
matrix: ${{ steps.set-matrix.outputs.matrix }}
45+
steps:
46+
- uses: actions/checkout@v2
47+
with:
48+
fetch-depth: 0
49+
- name: Generating build matrix
50+
id: set-matrix
51+
run: |
52+
# outputting for debugging purposes
53+
python scripts/github_actions/generate_build_matrix.py
54+
MATRIX=$(python scripts/github_actions/generate_build_matrix.py)
55+
echo "::set-output name=matrix::${MATRIX}"
56+
57+
build_conda_cpu_ubuntu:
58+
needs: generate_build_matrix
59+
runs-on: ubuntu-18.04
4260
strategy:
4361
fail-fast: false
4462
matrix:
45-
os: [ubuntu-18.04, macos-10.15]
46-
python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
47-
# from https://download.pytorch.org/whl/torch_stable.html
48-
#
49-
# PyTorch 1.11.x supports 3.7, 3.8, 3.9, 3.10
50-
# PyTorch 1.10, 1.9.x, 1.8.x, and 1.7.1 support 3.6, 3.7, 3.8, 3.9
51-
# PyTorch 1.7.0, 1.6.0, and 1.5.x support 3.6, 3.7, 3.8
52-
#
53-
# Other PyTorch versions are not tested
54-
#
55-
torch: ["1.5.0", "1.5.1", "1.6.0", "1.7.0", "1.7.1", "1.8.0", "1.8.1", "1.9.0", "1.9.1", "1.10.0", "1.10.1", "1.10.2", "1.11.0"]
56-
exclude:
57-
- python-version: "3.9" # exclude Python 3.9 for [1.5.0, 1.5.1, 1.6.0, 1.7.0]
58-
torch: "1.5.0"
59-
- python-version: "3.9"
60-
torch: "1.5.1"
61-
- python-version: "3.9"
62-
torch: "1.6.0"
63-
- python-version: "3.9"
64-
torch: "1.7.0"
65-
- python-version: "3.10" # exclude Python 3.10 for [1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2]
66-
torch: "1.5.0"
67-
- python-version: "3.10"
68-
torch: "1.5.1"
69-
- python-version: "3.10"
70-
torch: "1.6.0"
71-
- python-version: "3.10"
72-
torch: "1.7.0"
73-
- python-version: "3.10"
74-
torch: "1.7.1"
75-
- python-version: "3.10"
76-
torch: "1.8.0"
77-
- python-version: "3.10"
78-
torch: "1.8.1"
79-
- python-version: "3.10"
80-
torch: "1.9.0"
81-
- python-version: "3.10"
82-
torch: "1.9.1"
83-
- python-version: "3.10"
84-
torch: "1.10.0"
85-
- python-version: "3.10"
86-
torch: "1.10.1"
87-
- python-version: "3.10"
88-
torch: "1.10.2"
89-
- python-version: "3.6" # exclude Python 3.6 for [1.11.0]
90-
torch: "1.11.0"
63+
${{ fromJson(needs.generate_build_matrix.outputs.matrix) }}
9164

9265
steps:
9366
# refer to https://github.com/actions/checkout
@@ -134,25 +107,11 @@ jobs:
134107
./scripts/build_conda_cpu.sh
135108
136109
- name: Display generated files
137-
if: startsWith(matrix.os, 'ubuntu')
138110
run: |
139111
ls -lh /usr/share/miniconda/envs/k2/conda-bld/linux-64
140112
141-
- name: Display generated files
142-
if: startsWith(matrix.os, 'macos')
143-
run: |
144-
ls -lh /usr/local/miniconda/envs/k2/conda-bld/osx-64
145-
146113
- name: Upload generated files
147-
if: startsWith(matrix.os, 'ubuntu')
148114
uses: actions/upload-artifact@v2
149115
with:
150116
name: torch-${{ matrix.torch }}-python-${{ matrix.python-version }}-${{ matrix.os }}
151117
path: /usr/share/miniconda/envs/k2/conda-bld/linux-64/*.tar.bz2
152-
153-
- name: Upload generated files
154-
if: startsWith(matrix.os, 'macos')
155-
uses: actions/upload-artifact@v2
156-
with:
157-
name: torch-${{ matrix.torch }}-python-${{ matrix.python-version }}-${{ matrix.os }}
158-
path: /usr/local/miniconda/envs/k2/conda-bld/osx-64/*.tar.bz2

‎.github/workflows/windows-conda.yml ‎.github/workflows/build-conda-cpu-windows.yml

+24-18
Original file line numberDiff line numberDiff line change
@@ -15,36 +15,42 @@
1515
# limitations under the License.
1616

1717

18-
name: build-windows-conda
18+
name: build_conda_cpu_windows
1919

2020
on:
2121
push:
22-
branches:
23-
- conda-win
22+
tags:
23+
- '*'
2424

2525
env:
2626
BUILD_TYPE: Release
2727

2828
jobs:
29-
build-windows-conda:
29+
generate_build_matrix:
30+
# see https://github.com/pytorch/pytorch/pull/50633
31+
runs-on: ubuntu-latest
32+
outputs:
33+
matrix: ${{ steps.set-matrix.outputs.matrix }}
34+
steps:
35+
- uses: actions/checkout@v2
36+
with:
37+
fetch-depth: 0
38+
- name: Generating build matrix
39+
id: set-matrix
40+
run: |
41+
# outputting for debugging purposes
42+
python scripts/github_actions/generate_build_matrix.py
43+
MATRIX=$(python scripts/github_actions/generate_build_matrix.py)
44+
echo "::set-output name=matrix::${MATRIX}"
45+
46+
build_conda_cpu_windows:
3047
# see https://github.com/actions/virtual-environments/blob/win19/20210525.0/images/win/Windows2019-Readme.md
31-
runs-on: ${{ matrix.os }}
48+
needs: generate_build_matrix
49+
runs-on: windows-2019
3250
strategy:
3351
fail-fast: false
3452
matrix:
35-
os: [windows-2019]
36-
# Python 3.9 is for PyTorch 1.7.1, 1.8.x, 1.9.0,
37-
python-version: [3.6, 3.7, 3.8, 3.9]
38-
torch: ["1.5.0", "1.5.1", "1.6.0", "1.7.0", "1.7.1", "1.8.0", "1.8.1", "1.9.0"]
39-
exclude:
40-
- python-version: 3.9 # exclude Python 3.9 for [1.5.0, 1.5.1, 1.6.0, 1.7.0]
41-
torch: "1.5.0"
42-
- python-version: 3.9
43-
torch: "1.5.1"
44-
- python-version: 3.9
45-
torch: "1.6.0"
46-
- python-version: 3.9
47-
torch: "1.7.0"
53+
${{ fromJson(needs.generate_build_matrix.outputs.matrix) }}
4854

4955
steps:
5056
- uses: actions/checkout@v2

0 commit comments

Comments
 (0)