Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.804219697s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:20:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.804243432s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X.......................................XX........X....X.] duration: 25.010089077s trace: https://jaeger.dev.storj.io/trace/1818869bcdf5236f FAIL (2026-01-16 11:20:57 UTC) orbiter warning: context deadline exceeded Duration: 25.010115607s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.625452313s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:20:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.62549817s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..X.......................................XX........X....X] duration: 30.000485167s trace: https://jaeger.dev.storj.io/trace/1818869bcdf5236f FAIL (2026-01-16 11:20:21 UTC) orbiter warning: context deadline exceeded Duration: 30.000496048s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXXXXXXXXX] duration: 5.000988983s trace: https://jaeger.dev.storj.io/trace/5baaa6c5fd4b59fc FAIL (2026-01-16 11:20:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001012087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.725571016s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:20:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.725598008s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X.......................................XX........X....] duration: 18.668162238s trace: https://jaeger.dev.storj.io/trace/166bdf9b2f21d261 FAIL (2026-01-16 11:19:42 UTC) orbiter warning: context deadline exceeded Duration: 18.668186483s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.757868111s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:19:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.757896435s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [..X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXXXXXXXX] duration: 5.001220041s trace: https://jaeger.dev.storj.io/trace/64beff00e6c83b5d FAIL (2026-01-16 11:19:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001241373s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X.......................................XX........X...] duration: 13.966493115s trace: https://jaeger.dev.storj.io/trace/166bdf9b2f21d261 FAIL (2026-01-16 11:18:58 UTC) orbiter warning: context deadline exceeded Duration: 13.966519114s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.829831633s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:18:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.829857133s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X.......................................XX........X..] duration: 15.447785645s trace: https://jaeger.dev.storj.io/trace/166bdf9b2f21d261 FAIL (2026-01-16 11:18:36 UTC) orbiter warning: context deadline exceeded Duration: 15.447812687s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.662201314s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:18:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.662227495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [...X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXXXXXXX] duration: 5.001501989s trace: https://jaeger.dev.storj.io/trace/170dc92a295c6484 FAIL (2026-01-16 11:18:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001512189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X.......................................XX........X.] duration: 21.517700217s trace: https://jaeger.dev.storj.io/trace/166bdf9b2f21d261 FAIL (2026-01-16 11:18:02 UTC) orbiter warning: context deadline exceeded Duration: 21.51771791s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X.......................................XX........X] duration: 30.000870972s trace: https://jaeger.dev.storj.io/trace/166bdf9b2f21d261 FAIL (2026-01-16 11:17:41 UTC) orbiter warning: context deadline exceeded Duration: 30.000891733s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXXXXXX] duration: 5.001424984s trace: https://jaeger.dev.storj.io/trace/6b04827d043b54a1 FAIL (2026-01-16 11:17:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001453047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.672527026s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:17:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.672557175s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXXXXX] duration: 5.001526186s trace: https://jaeger.dev.storj.io/trace/3329b9fe6d82b726 FAIL (2026-01-16 11:16:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001548258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X..X.......................................XX........] duration: 9.849574842s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:16:41 UTC) orbiter warning: context deadline exceeded Duration: 9.849613595s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.804728806s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.804745548s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXXXX] duration: 5.001475631s trace: https://jaeger.dev.storj.io/trace/595c4181aab3d173 FAIL (2026-01-16 11:15:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00149608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..X.......................................XX.......] duration: 24.547611341s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:15:50 UTC) orbiter warning: context deadline exceeded Duration: 24.547633222s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.658504897s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:15:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.658527529s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXXX] duration: 5.002052025s trace: https://jaeger.dev.storj.io/trace/3a273cfbda1b456c FAIL (2026-01-16 11:15:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002144904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X..X.......................................XX......] duration: 18.493170693s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:15:25 UTC) orbiter warning: context deadline exceeded Duration: 18.493194318s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.678602194s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:15:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.678629506s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X..X.......................................XX.....] duration: 18.33058989s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:14:50 UTC) orbiter warning: context deadline exceeded Duration: 18.330650215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.653949012s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:14:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.653987496s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XX.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXXX] duration: 5.001776911s trace: https://jaeger.dev.storj.io/trace/1fd424d55b4e7847 FAIL (2026-01-16 11:14:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001789625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X..X.......................................XX....] duration: 8.255624303s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:14:10 UTC) orbiter warning: context deadline exceeded Duration: 8.255644903s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X........X..X.......................................XX...] duration: 29.922998401s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:14:02 UTC) orbiter warning: context deadline exceeded Duration: 29.923028778s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.75400755s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:13:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.754031176s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXX.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XXX] duration: 5.00061464s trace: https://jaeger.dev.storj.io/trace/21a551356f3c7bbd FAIL (2026-01-16 11:13:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000628676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X........X..X.......................................XX..] duration: 17.659027168s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:13:12 UTC) orbiter warning: context deadline exceeded Duration: 17.65905405s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.701951141s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:12:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.701979635s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.XXX.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.XX] duration: 5.001348296s trace: https://jaeger.dev.storj.io/trace/7579361995a04408 FAIL (2026-01-16 11:12:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001367662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X....X........X..X.......................................XX.] duration: 18.849536714s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:12:25 UTC) orbiter warning: context deadline exceeded Duration: 18.849561521s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.732086272s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:12:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.732108234s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X....X........X..X.......................................XX] duration: 30.001675535s trace: https://jaeger.dev.storj.io/trace/41bb1a1ace9a9b7e FAIL (2026-01-16 11:11:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001686075s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [X.XXX.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.X] duration: 5.0013596s trace: https://jaeger.dev.storj.io/trace/7c11ce1c95735c86 FAIL (2026-01-16 11:11:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001382442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.721069914s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:11:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 5.721095464s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XX.XXX.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX.] duration: 4.198114658s trace: https://jaeger.dev.storj.io/trace/3687c43b601b43b1 FAIL (2026-01-16 11:11:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.198131429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....X........X..X.......................................X] duration: 30.001240179s trace: https://jaeger.dev.storj.io/trace/324c82cba672933d FAIL (2026-01-16 11:10:52 UTC) orbiter warning: context deadline exceeded Duration: 30.001260157s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 27.57580169s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:10:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 27.575830534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003137773s trace: https://jaeger.dev.storj.io/trace/49d11cc9e6224d96 FAIL (2026-01-16 11:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 11:09:47.771372914 (19h13m44.665081533s) [upload=16894]: Starting upload Jan 16 11:09:47.771479847 (19h13m44.665188456s) [upload=16894]: Done starting upload Jan 16 11:09:47.771531725 (19h13m44.665240335s) [upload=16894]: Uploading segments... Jan 16 11:09:47.774149089 (19h13m44.667857698s) [upload=16894]: Got next segment. Inline: false Jan 16 11:09:47.774212470 (19h13m44.667921089s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 11:09:47.774228811 (19h13m44.66793742s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 11:09:48.269059003 (19h13m45.162767612s) [seg_pos={0 0}, upload=16894]: Begin upload segment... Jan 16 11:09:48.269110061 (19h13m45.16281867s) [seg_pos={0 0}, upload=16894]: Done begin upload segment. Jan 16 11:09:48.269126592 (19h13m45.162835201s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269203779 (19h13m45.162912648s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.269884625 (19h13m45.163593235s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270218901 (19h13m45.16392751s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270496871 (19h13m45.16420548s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.270755604 (19h13m45.164464213s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271002003 (19h13m45.164710612s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271283910 (19h13m45.164992519s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.271614479 (19h13m45.165323088s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272775841 (19h13m45.16648445s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.272802050 (19h13m45.166510659s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273863231 (19h13m45.16757184s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.273981255 (19h13m45.167689865s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275103192 (19h13m45.168811801s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.275142927 (19h13m45.168851536s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:48.276824178 (19h13m45.170532787s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Uploading piece... Jan 16 11:09:50.068745560 (19h13m46.962454169s) [upload=16894]: Next returned nil segment Jan 16 11:09:50.068765678 (19h13m46.962474287s) [upload=16894]: Waiting for error group managing segments... Jan 16 11:09:50.406014166 (19h13m47.299722785s) [noise=true , address=77.90.31.19:4841 , piece_id=T6NNJTRIS32MYHEPCANSHDUHRA7ETOEXX4Z4JUYTUY4UCM3RTGSQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.413138982 (19h13m47.306847612s) [noise=true , address=77.90.31.15:4700 , piece_id=CCEDNAYUKEKSZPNFBRNOWZVMS4G4PJ6HTMLVGBLT63UNC4LRU4UQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.414463655 (19h13m47.308172274s) [noise=true , address=77.90.31.11:4555 , piece_id=6NVZ7OKS7HK3IOY5QHJMJXIKBG6NKFG2TUWUG3TCCRLGJ2F4S6RQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.438348053 (19h13m47.332056672s) [noise=true , address=145.79.227.34:4905 , piece_id=OFIH5I5FTWRSK7IMG6JTDAZRQ7QFS3A5VWICISG763GDFJR35EVA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.447803696 (19h13m47.341512305s) [noise=true , address=145.79.227.2:4113 , piece_id=HNJPCWABGS7F7O46WPQVE7CNVZKR2W757CRKBFNGJ3WNCQ5N7CAA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.491598103 (19h13m47.385306712s) [noise=true , address=145.79.227.41:4086 , piece_id=EWAJFX7M5THWKIF2F5TH7L6VKZPZKLX2JHF3IGOIR344PJQD3FXQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.510437013 (19h13m47.404145622s) [noise=true , address=145.79.227.36:4518 , piece_id=QI2HN6JYG7KJ7JUQWL7QGBJ7NGAKFFYR4QX5B7V2AO4XAVH5CZLA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516143880 (19h13m47.409852489s) [noise=true , address=145.79.227.4:4186 , piece_id=WQIAYDK2CJVWVNJAAMSMFEUT3IWPUKVSPRE45F2MR3HJTWWHZVZA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.516602503 (19h13m47.410311112s) [noise=true , address=45.91.116.13:4154 , piece_id=A2J4JHHOIKRXCMF3N62JYQQZOKQQCHO4IT3YXPQKCVUU5W5CSQAQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.526149721 (19h13m47.41985833s) [noise=true , address=145.79.227.29:4941 , piece_id=3S7RQB7A236MCEZC3DXLZEMYBO2V42SEZBWEJFPTLN45QNOPIP5A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.530495044 (19h13m47.424203654s) [noise=true , address=45.91.116.11:4117 , piece_id=4UAANAX5DIZJQ3VXOSAE6RECZORUHOADVKI6RW54KUALK2EV5GJA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.546127160 (19h13m47.439835769s) [noise=true , address=45.91.116.6:4975 , piece_id=XRCEJQJYBEDIP43BUQNDJFBHUBV4UCOMCJQC43YQEFQAGNUPDOQQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628524367 (19h13m47.522232976s) [noise=true , address=45.91.116.5:4297 , piece_id=GRMFXRMWR7QTIKQLMMTHJVEUQPDFWSNWQSX3JRHT3JBLH5NPZTBQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: <nil> Jan 16 11:09:50.628568943 (19h13m47.522277552s) [seg_pos={0 0}, upload=16894]: Satisfied cohort requirements after 13 pieces Jan 16 11:09:50.628865046 (19h13m47.522573655s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Jan 16 11:09:50.628871859 (19h13m47.522580468s) [noise=true , address=145.79.227.39:4013 , piece_id=CWRY72DBHVGJWQAW5HWZEJ6X53DHKWJUZZFRAXMAH63SLUKH45XQ, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629045590 (19h13m47.522754199s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:45166->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629061130 (19h13m47.522769739s) [noise=true , address=77.90.31.21:4877 , piece_id=KSCPWZBJZ2P5SRTA7ZOD7KUTDYWTCRRKWC74YCNHM2C3LLLI7E6A, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629205314 (19h13m47.522913923s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: read tcp 188.166.225.170:39444->145.79.227.35:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 11:09:50.629211035 (19h13m47.522919644s) [noise=true , address=145.79.227.35:4483 , piece_id=2B5QCYCWKC6NHXJQITWVXJPZWNJL4XJRS3OU2IMTHIDL5F3FGXBA, seg_pos={0 0}, upload=16894]: Long tail cancellation due to optimal threshold Jan 16 11:09:50.629229471 (19h13m47.52293808s) [seg_pos={0 0}, upload=16894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 11:09:50.629299975 (19h13m47.523008584s) [upload=16894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 11:09:50.629304894 (19h13m47.523013503s) [upload=16894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 11:09:50.940599124 (19h13m47.834307733s) [upload=16894]: Done uploading segments... Jan 16 11:09:50.940620665 (19h13m47.834329274s) [upload=16894]: Upload finished. err: <nil> Duration: 30.003168591s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXX.XXX.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XXX] duration: 5.001313028s trace: https://jaeger.dev.storj.io/trace/3687c43b601b43b1 FAIL (2026-01-16 11:09:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001346272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X....X........X..X.......................................] duration: 19.602402674s trace: https://jaeger.dev.storj.io/trace/633ecb0cc3d3afc9 FAIL (2026-01-16 11:09:52 UTC) orbiter warning: context deadline exceeded Duration: 19.602452268s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXX.XXX.X....X...X..XXXXX.XXXXX.XXXXXXXXXXXXXXXXXX.XXXXX.XX] duration: 5.002048979s trace: https://jaeger.dev.storj.io/trace/48e0079c47207339 FAIL (2026-01-16 11:09:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002071403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X....X........X..X......................................] duration: 29.529266335s trace: https://jaeger.dev.storj.io/trace/633ecb0cc3d3afc9 FAIL (2026-01-16 11:09:15 UTC) orbiter warning: context deadline exceeded Duration: 29.529305479s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)