Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X............X...................X..X.....X...........] duration: 15.231677795s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:53:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.231702893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......XX........X.......................XX...............] duration: 15.439623073s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:53:40 UTC) orbiter warning: context deadline exceeded Duration: 15.439672547s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.......XX........X.......................XX..............] duration: 27.933842611s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:53:23 UTC) orbiter warning: context deadline exceeded Duration: 27.933867578s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001704608s trace: https://jaeger.dev.storj.io/trace/736a7a3d4d3c8b4b FAIL (2026-01-12 16:53:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00176923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...................X............................] duration: 5.998558353s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:52:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.998598239s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X............X...................X..X.....X..........] duration: 18.941249441s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:52:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.941274147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.......XX........X.......................XX.............] duration: 28.8540062s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:52:31 UTC) orbiter warning: context deadline exceeded Duration: 28.854032378s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...................X...........................] duration: 5.887717732s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:52:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.887740955s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.00294065s trace: https://jaeger.dev.storj.io/trace/796732885d2c796a FAIL (2026-01-12 16:52:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002962892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...................X..........................] duration: 5.871174055s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:51:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.871204382s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000747585s trace: https://jaeger.dev.storj.io/trace/30c45567bc316cf8 FAIL (2026-01-12 16:51:33 UTC) orbiter warning: context deadline exceeded Duration: 5.000772271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.......XX........X.......................XX............] duration: 16.002565705s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:51:25 UTC) orbiter warning: context deadline exceeded Duration: 16.002616892s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X............X...................X..X.....X.........] duration: 13.028013552s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:51:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.028043288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.......XX........X.......................XX...........] duration: 28.223514963s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:51:00 UTC) orbiter warning: context deadline exceeded Duration: 28.223531665s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...................X.........................] duration: 5.855455433s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:50:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.855485019s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........X............X...................X..X.....X........] duration: 13.29644074s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:50:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.296456179s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....X.......XX........X.......................XX..........] duration: 28.911302664s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:50:31 UTC) orbiter warning: context deadline exceeded Duration: 28.911330898s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000720008s trace: https://jaeger.dev.storj.io/trace/5caa6fa9b7976d37 FAIL (2026-01-12 16:50:20 UTC) orbiter warning: context deadline exceeded Duration: 5.000732652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........X............X...................X..X.....X.......] duration: 30.744876981s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:50:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.74489663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...................X........................] duration: 5.86315385s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:50:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.86317986s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.....X.......XX........X.......................XX.........] duration: 16.022885392s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:49:58 UTC) orbiter warning: context deadline exceeded Duration: 16.02291046s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.....X.......XX........X.......................XX........] duration: 15.748871646s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:49:35 UTC) orbiter warning: context deadline exceeded Duration: 15.748886275s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........X............X...................X..X.....X......] duration: 21.576210583s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:49:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.576240259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...................X.......................] duration: 5.896866832s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:49:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.896885879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000835928s trace: https://jaeger.dev.storj.io/trace/1d06b7b922f2511 FAIL (2026-01-12 16:49:29 UTC) orbiter warning: context deadline exceeded Duration: 5.000848534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.....X.......XX........X.......................XX.......] duration: 18.885979112s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:49:19 UTC) orbiter warning: context deadline exceeded Duration: 18.886005944s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...................X......................] duration: 5.799210259s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:48:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.799237892s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X...........X............X...................X..X.....X.....] duration: 15.087874952s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:48:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.087901122s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXXX] duration: 5.001111985s trace: https://jaeger.dev.storj.io/trace/6c38245428ec233b FAIL (2026-01-12 16:48:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001136381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.....X.......XX........X.......................XX......] duration: 19.353277143s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:48:08 UTC) orbiter warning: context deadline exceeded Duration: 19.353300558s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...................X.....................] duration: 6.167749067s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:47:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 6.167775987s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.....X.......XX........X.......................XX.....] duration: 21.809329534s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:47:21 UTC) orbiter warning: context deadline exceeded Duration: 21.809353719s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X...........X............X...................X..X.....X....] duration: 17.034368684s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:47:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.034399482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXXX] duration: 5.001376417s trace: https://jaeger.dev.storj.io/trace/597bcf8e5f97fa7b FAIL (2026-01-12 16:47:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001399843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...................X....................] duration: 5.789777126s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:46:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.789808096s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.....X.......XX........X.......................XX....] duration: 20.487335915s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:46:40 UTC) orbiter warning: context deadline exceeded Duration: 20.487400978s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X.X...........X............X...................X..X.....X...] duration: 13.463510641s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:46:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.463544446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...................X...................] duration: 5.783776501s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:46:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.78380685s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXXX] duration: 5.00125669s trace: https://jaeger.dev.storj.io/trace/1b8666b101e89a07 FAIL (2026-01-12 16:46:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001279194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X.....X.......XX........X.......................XX...] duration: 18.411752539s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:46:03 UTC) orbiter warning: context deadline exceeded Duration: 18.411778238s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXXX] duration: 5.000927533s trace: https://jaeger.dev.storj.io/trace/1fd4e725c29dd9cb FAIL (2026-01-12 16:45:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00094722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...................X..................] duration: 5.802241971s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:45:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.802265245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X.X...........X............X...................X..X.....X..] duration: 28.80564115s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:45:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.805656939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X.....X.......XX........X.......................XX..] duration: 19.977618474s trace: https://jaeger.dev.storj.io/trace/79bec5398380a434 FAIL (2026-01-12 16:45:12 UTC) orbiter warning: context deadline exceeded Duration: 19.9776738s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXXX] duration: 5.000562518s trace: https://jaeger.dev.storj.io/trace/4fb70b38b1cb5eb5 FAIL (2026-01-12 16:45:07 UTC) orbiter warning: context deadline exceeded Duration: 5.000590422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXXX] duration: 5.001134348s trace: https://jaeger.dev.storj.io/trace/2ba45d86177c96ae FAIL (2026-01-12 16:44:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001154526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...................X.................] duration: 5.874685336s trace: https://jaeger.dev.storj.io/trace/66a32f39f80f292d FAIL (2026-01-12 16:44:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 16:30:55.249915044 (913h1m55.120998089s) [upload=799132]: Starting upload Jan 12 16:30:55.249987422 (913h1m55.121070477s) [upload=799132]: Done starting upload Jan 12 16:30:55.250028591 (913h1m55.121111636s) [upload=799132]: Uploading segments... Jan 12 16:30:55.250211880 (913h1m55.121294925s) [upload=799132]: Got next segment. Inline: false Jan 12 16:30:55.250245594 (913h1m55.121328629s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:30:55.250251425 (913h1m55.12133446s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:30:56.071937025 (913h1m55.94302007s) [seg_pos={0 0}, upload=799132]: Begin upload segment... Jan 12 16:30:56.071976670 (913h1m55.943059715s) [seg_pos={0 0}, upload=799132]: Done begin upload segment. Jan 12 16:30:56.071998722 (913h1m55.943081758s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072083724 (913h1m55.943166759s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.072657847 (913h1m55.943740882s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073108194 (913h1m55.944191239s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073496294 (913h1m55.944579339s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.073940249 (913h1m55.945023294s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074379576 (913h1m55.945462611s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074394413 (913h1m55.945477459s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.074831386 (913h1m55.945914431s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075189457 (913h1m55.946272493s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.075651467 (913h1m55.946734502s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076095152 (913h1m55.947178197s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.076518047 (913h1m55.947601092s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.077884248 (913h1m55.948967283s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.086222816 (913h1m55.957305861s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:56.087463598 (913h1m55.958546633s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Uploading piece... Jan 12 16:30:57.919337452 (913h1m57.790420487s) [upload=799132]: Next returned nil segment Jan 12 16:30:57.919357399 (913h1m57.790440444s) [upload=799132]: Waiting for error group managing segments... Jan 12 16:30:58.260660806 (913h1m58.131743851s) [noise=true , address=77.90.31.11:4555 , piece_id=CXD2F5JFINOROFC7UKNE3AZDBRKXSK2CP7OLYBWTRWVBLQWRJHVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.308291649 (913h1m58.179374704s) [noise=true , address=145.79.227.29:4941 , piece_id=JGAAXLAD6VOENF3E2FZX76KKRUVPC7ZFCVBTPFYXP5E22ZXV5BWA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.351167307 (913h1m58.222250342s) [noise=true , address=45.91.116.4:4939 , piece_id=VP2DATGUMIYLQTMM2PR4QICYGEKA44CIZWCQM2O256L4OOV3AAHA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.399158276 (913h1m58.270241311s) [noise=true , address=145.79.227.27:4691 , piece_id=B3UGNQGJ5ZZ24E7P453R2XYJDRDBYKXXRVZQV2GWNXCMGPTFJACQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.501634098 (913h1m58.372717143s) [noise=true , address=145.79.227.34:4906 , piece_id=2MNBKQ43LSRUUQBMMTMJGSSROUAXNTWIKU2OQOT5PRC2DCUJEXJA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.784125231 (913h1m58.655208276s) [noise=true , address=45.91.116.6:4976 , piece_id=ILMHKVME4HRASKCPYD56WAH2DHVOCW73WSDHSVPN4QKQZD3XJRUQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.836976424 (913h1m58.708059459s) [noise=true , address=77.90.31.12:4589 , piece_id=BRQM7DVHLLVYG37RMWE2LYWFG5S74NIFML7ATD5DB4DNIMZRYBQA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:58.920812865 (913h1m58.7918959s) [noise=true , address=77.90.31.13:4629 , piece_id=OIZSP7KV3RTYZNI3Y36XIMDMMYKLMO2YKC76PW2XT5XFTLSDGZGA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.028570624 (913h1m58.899653659s) [noise=true , address=45.91.116.12:4226 , piece_id=KSETR52IVCJTIFNSG2Z42O3K4GVCVVBFYAA35TGGVD6P3HDZ3MQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.266099516 (913h1m59.137182571s) [noise=true , address=77.90.31.15:4697 , piece_id=4B5A2U7IGUUDWD6B3FLQNF4MIMLJOPOL4AUC6RGYEEVIIY34RSKA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.584105403 (913h1m59.455188438s) [noise=true , address=145.79.227.30:4762 , piece_id=52VN3FLBHIBIFHA7W2HC5S5HBKRWI5OPYBGUVEYRINU2TC4MHOQQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:30:59.961408756 (913h1m59.832491791s) [noise=true , address=77.90.31.14:4661 , piece_id=EQCWMB2IFX6JQ4FKBJW5VP6LSI2GQPRI7SKR5HKI46SBE42VGYSA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386663834 (913h2m0.257746869s) [noise=true , address=77.90.31.16:4735 , piece_id=Q4U2JOHJMRKR5TVD7Z3I5OFAGBWG6JDSXW54KJUNDMBOIA4MC6GA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: <nil> Jan 12 16:31:00.386687379 (913h2m0.257770414s) [seg_pos={0 0}, upload=799132]: Segment reached optimal threshold of 13 pieces. Jan 12 16:31:00.386930361 (913h2m0.258013406s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45048->77.90.31.10:4520: use of closed network connection Jan 12 16:31:00.386937215 (913h2m0.25802025s) [noise=true , address=77.90.31.10:4520 , piece_id=VS6QXKOTR5NHRPDU3UBT64WIYOV7RVYYG5WCNA32GMKB4QJI4KVA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.386942885 (913h2m0.258025941s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 12 16:31:00.386953125 (913h2m0.25803616s) [noise=true , address=145.79.227.50:4373 , piece_id=CMCQX7EW4LA6BTHYD7HJJ34F4PN55CU3WGE4BYRO6LXMD7GUISZQ, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387241023 (913h2m0.258324058s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: context canceled; context canceled Jan 12 16:31:00.387249399 (913h2m0.258332434s) [noise=true , address=77.90.31.19:4842 , piece_id=PBDTJHEXY6ZPWK6LVWRRYX222YLAUOC3KWFDW5CZGNADPQYOZLCA, seg_pos={0 0}, upload=799132]: Long tail cancellation due to optimal threshold Jan 12 16:31:00.387265680 (913h2m0.258348715s) [seg_pos={0 0}, upload=799132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:31:00.387338268 (913h2m0.258421313s) [upload=799132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:31:00.387357725 (913h2m0.258440761s) [upload=799132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:31:01.427810625 (913h2m1.29889366s) [upload=799132]: Done uploading segments... Jan 12 16:31:01.427831484 (913h2m1.298914519s) [upload=799132]: Upload finished. err: <nil> Duration: 5.874704233s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X.X...........X............X...................X..X.....X.] duration: 19.766922978s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:44:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.766935242s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X.X...........X............X...................X..X.....X] duration: 1m0.001094033s trace: https://jaeger.dev.storj.io/trace/43e32d5953387a9 FAIL (2026-01-12 16:44:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001118049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX..XXXXXXXXXXXXXXXXXXXXXX.XX.XX.XXXXXXXXXXXXXXXX] duration: 5.000525789s trace: https://jaeger.dev.storj.io/trace/3471f3c06803044c FAIL (2026-01-12 16:44:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000548853s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)