Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 867574 reports (Page 11638 of 17352)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X.......X.....X............X.X.XX.....................X] duration: 5.001591405s trace: https://jaeger.dev.storj.io/trace/770b5381f98e6f64 FAIL (2025-12-09 05:11:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001612826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X..X......X................X.......................] duration: 5.740506906s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:11:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.740536802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.764325706s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.764359321s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................................X......] duration: 6.057648787s trace: https://jaeger.dev.storj.io/trace/7f5385cb64fb69 FAIL (2025-12-09 05:11:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.057687612s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.986425018s trace: https://jaeger.dev.storj.io/trace/2127b89fc0dfe472 FAIL (2025-12-09 05:10:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.986453855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................................X.....] duration: 6.226323505s trace: https://jaeger.dev.storj.io/trace/7f5385cb64fb69 FAIL (2025-12-09 05:10:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.226353192s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X.......X.....X............X.X.XX.....................] duration: 3.515372154s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:10:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515522891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X..X......X................X......................] duration: 5.781265844s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:10:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.78130075s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......X.......X.....X............X.X.XX....................] duration: 3.661498558s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661530067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.154840471s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:09:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.154867903s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.000829848s trace: https://jaeger.dev.storj.io/trace/2127b89fc0dfe472 FAIL (2025-12-09 05:09:47 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.001047472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................................X....] duration: 5.737279968s trace: https://jaeger.dev.storj.io/trace/7f5385cb64fb69 FAIL (2025-12-09 05:09:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.737298833s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X..X......X................X.....................] duration: 5.734779014s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:09:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.734806596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.069306412s trace: https://jaeger.dev.storj.io/trace/2127b89fc0dfe472 FAIL (2025-12-09 05:09:05 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.069339856s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.774296354s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:09:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.774365676s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........X.......X.....X............X.X.XX...................] duration: 3.463061717s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:08:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463096844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.111425331s trace: https://jaeger.dev.storj.io/trace/2127b89fc0dfe472 FAIL (2025-12-09 05:08:43 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.111452774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................................X...] duration: 5.960990852s trace: https://jaeger.dev.storj.io/trace/7f5385cb64fb69 FAIL (2025-12-09 05:08:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.961021741s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X..X......X................X....................] duration: 5.854476922s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:08:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.854508834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.983708406s trace: https://jaeger.dev.storj.io/trace/2127b89fc0dfe472 FAIL (2025-12-09 05:08:17 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.98374234s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X.......X.....X............X.X.XX..................] duration: 3.45010449s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:08:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450130849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.447738314s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:08:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.447769153s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000961709s trace: https://jaeger.dev.storj.io/trace/2127b89fc0dfe472 FAIL (2025-12-09 05:07:58 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 30.000984653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X..X......X................X...................] duration: 5.869739185s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.8697632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................................X..] duration: 5.91939487s trace: https://jaeger.dev.storj.io/trace/7f5385cb64fb69 FAIL (2025-12-09 05:07:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.919422914s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.20364266s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:06:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.203808484s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........X.......X.....X............X.X.XX.................] duration: 3.40306579s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:06:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403117599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.................................X.] duration: 6.014880615s trace: https://jaeger.dev.storj.io/trace/7f5385cb64fb69 FAIL (2025-12-09 05:06:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.014899159s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.................................X] duration: 9m50.138286816s trace: https://jaeger.dev.storj.io/trace/7f5385cb64fb69 FAIL (2025-12-09 05:06:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9m50.138893583s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X..X......X................X..................] duration: 5.641967655s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:06:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.641998625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.677287815s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.677317463s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........X.......X.....X............X.X.XX................] duration: 3.52350438s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:06:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523539366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.600703019s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:05:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.600733637s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X..............X..X......X................X.................] duration: 5.737687898s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:05:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.737716601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.313503524s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:05:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.313532291s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [............X.......X.....X............X.X.XX...............] duration: 3.449419607s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:05:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449453471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.287638648s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:05:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.287666772s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X..............X..X......X................X................] duration: 5.696385702s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:04:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.696412234s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.............X.......X.....X............X.X.XX..............] duration: 3.448956161s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:04:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448985306s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 10.168620736s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:04:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.168692513s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.X..............X..X......X................X...............] duration: 5.626926434s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:04:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.627012428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.301650232s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:03:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.301678417s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..............X.......X.....X............X.X.XX.............] duration: 3.469740515s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:03:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469768247s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X.X..............X..X......X................X..............] duration: 5.821253358s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:03:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.821284176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.18871447s trace: https://jaeger.dev.storj.io/trace/56a3475297485c08 FAIL (2025-12-09 05:02:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== X5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065270864 (85h22m45.936353909s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.065752381 (85h22m45.936835426s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066247914 (85h22m45.937330959s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.066774447 (85h22m45.937857532s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067303855 (85h22m45.938386891s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.067822694 (85h22m45.938905739s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068333786 (85h22m45.939416881s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.068849508 (85h22m45.939932543s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.069469499 (85h22m45.940552544s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.072036628 (85h22m45.943119673s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Uploading piece... Dec 9 04:51:46.888693467 (85h22m46.759776502s) [upload=74556]: Next returned nil segment Dec 9 04:51:46.888707253 (85h22m46.759790288s) [upload=74556]: Waiting for error group managing segments... Dec 9 04:51:47.279773158 (85h22m47.150856203s) [noise=true , address=185.167.204.205:4020 , piece_id=3BUJEVVGL6UH65INUWX27BD5FGW6MB6DJ5SSBACQ4D5RQOHKT5BA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.282474582 (85h22m47.153557617s) [noise=true , address=185.167.204.205:4034 , piece_id=FTR5MFLENHZTTEW6L7NWFDIJGPTAZGWU6GP4DDDQDZK35ZGSCADA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.287117565 (85h22m47.15820061s) [noise=true , address=185.167.204.205:4007 , piece_id=GKRAWJPU5XDYMPPGHZOSOCF54MITRSUEZRWKYSGHFEJF2TC7YN6Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.288297912 (85h22m47.159380957s) [noise=true , address=185.167.204.206:4061 , piece_id=VUL4OSL3BTKXWWMZZ26JLOVTQ2BMQACPGBWHGP247C5AKGJS3HTA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289440658 (85h22m47.160523693s) [noise=true , address=185.167.204.206:4037 , piece_id=XGONHPKOKCPYQZATQY3Z4U2ZNNBV7YE2JRX5FF3ZCFHQEZBV7XSA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.289851421 (85h22m47.160934476s) [noise=true , address=185.167.204.205:4023 , piece_id=KL24TWTNUAJJALRGBH4KMEPC6PYGV7ONVN5LV64JD2V5PAR5D6CQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.290269507 (85h22m47.161352552s) [noise=true , address=185.167.204.205:4035 , piece_id=QOB3YZQVM3V4BGQWAMWLCGADGOB5Z7RDFGTB63SQGPNO25RGMIRA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.295536938 (85h22m47.166619983s) [noise=true , address=185.167.204.205:4009 , piece_id=TPSFBANPH3QMOETHSFBYIFKK3TTXNLLTK3EBUWD22EBWKTMZW6HQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300739596 (85h22m47.171822631s) [noise=true , address=185.167.204.205:4014 , piece_id=SKRUXMC2TQMNEZ6CYBZVMLYNB4K3X452W7M7EGVVFFSTOITCP3KA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.300896534 (85h22m47.171979579s) [noise=true , address=185.167.204.205:4010 , piece_id=RS6DJ2OJMOYHRJKERWHGGROKFV73RFFXJBM7PUXUKG5VOVQOGBPQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.303634699 (85h22m47.174717744s) [noise=true , address=185.167.204.206:4039 , piece_id=3OOA4F22GJOWTNXBJL34PZKL5BA372E4IEDFZEDXIAYB7LQFY2AA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.304032196 (85h22m47.175115241s) [noise=true , address=185.167.204.205:4011 , piece_id=J2YM4SX5MQXVPLTKCUWJQD3BNTQCJ3HZD667DRS6HKFVNY4OEWQQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.847206642 (85h22m47.718289677s) [noise=true , address=81.171.14.70:4025 , piece_id=NJUWMN24CHGRWP4OS2CPUW4HMYQK7VJYGE3GBRDRDZK76PSBT3KQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.878231424 (85h22m47.749314469s) [noise=true , address=81.171.14.70:4027 , piece_id=2YW52FPZHNG3TGLT32NSJDMRN2ZPTDID44U7BV73DPYHIGTLDWHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.913929187 (85h22m47.785012232s) [noise=true , address=81.171.14.70:4038 , piece_id=I3NZYZNGGEBDLOF5AW2WNFHAA3BDFIN3XREIZAV6W4R3UJ2KAOHQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:47.979060731 (85h22m47.850143776s) [noise=true , address=81.171.14.70:4046 , piece_id=7KWFQRAUKO2AQAPOVNM4FFZBKK3BITP7AGA4QBZ7T3SPXJVTVPEQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.290181480 (85h22m48.161264545s) [noise=true , address=81.171.14.70:4026 , piece_id=2SD4KRFMZ6VOLCIWFEMETCYLDX7GJZ5UYWO2VTWT3SLV47IWMIUA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.348307143 (85h22m48.219390188s) [noise=true , address=81.171.14.70:4036 , piece_id=CDLBWGBE7H66MYAFEG2SYVALAAWRZK5IBPLYWRQOEPOBYCQ3YHAA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.400755824 (85h22m48.27183886s) [noise=true , address=81.171.14.70:4042 , piece_id=YES6SNMMWSPGOJ6BBMYCFLLQS4EWEW22G52DAMPRIKBNJZESXP3Q, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.533382936 (85h22m48.404465981s) [noise=true , address=89.149.201.147:4004 , piece_id=CCCMJCLKJWELUFRR7M74HQ2C5AJDUZQNKCK54QMQUHEFWKY3T46A, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.553385017 (85h22m48.424468052s) [noise=true , address=89.149.201.147:4021 , piece_id=74TPYB6LHKHFF6K5QD6B5HQNMXGUY5A7UO6TRXWGDTKBUMTFC7JQ, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.554766448 (85h22m48.425849483s) [noise=true , address=89.149.201.147:4016 , piece_id=7HGY5QZTO35LFESAPMT4VSSHOM65KDIW3MZJLZ42C5OBK22U4STA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.567049941 (85h22m48.438132986s) [noise=true , address=89.149.201.147:4009 , piece_id=D6CNWDB4I5X64SCJ74KWYDZH4XXGDI3GVSL6NXHZWMSH6EJN3QJA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.609431201 (85h22m48.480514246s) [noise=true , address=89.149.201.147:4005 , piece_id=IX6QMA7PZSNDM535TNPWI4S5OMHPF4NAKEY3SEMZOZTJDYCYRTPA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.946626053 (85h22m48.817709098s) [noise=true , address=89.149.201.147:4002 , piece_id=SAENLJHQ4VDGI66VIAPRYGICLC6YUJWQYGDXY7OGOTLFGE2E6KFA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979683347 (85h22m48.850766393s) [noise=true , address=89.149.201.147:4013 , piece_id=LYQZQTEIZIFJC7R5WXXLBREQ4VIN5MQRKZRW4FJ42WI62WKMDNYA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: <nil> Dec 9 04:51:48.979705049 (85h22m48.850788094s) [seg_pos={0 0}, upload=74556]: Segment reached optimal threshold of 26 pieces. Dec 9 04:51:48.979913476 (85h22m48.850996511s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.979921591 (85h22m48.851004636s) [noise=true , address=89.149.201.147:4007 , piece_id=TD6VZFCIEFULPTYHOEYJS5BQM2BIP4IG4UUU5L7OITHQQSWWTGHA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980162019 (85h22m48.851245064s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 9 04:51:48.980173240 (85h22m48.851256285s) [noise=true , address=89.149.201.147:4023 , piece_id=MSQYWELFCDSYKRTQGD66X3IEKPLHGVO3SAXAOTNUO2GBP2GBVASA, seg_pos={0 0}, upload=74556]: Long tail cancellation due to optimal threshold Dec 9 04:51:48.980194180 (85h22m48.851277216s) [seg_pos={0 0}, upload=74556]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 04:51:48.980337884 (85h22m48.851420929s) [upload=74556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 04:51:48.980346260 (85h22m48.851429295s) [upload=74556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 04:53:36.645778379 (85h24m36.516861414s) [upload=74556]: Done uploading segments... Dec 9 04:53:36.645916613 (85h24m36.516999678s) [upload=74556]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.188742323s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...............X.......X.....X............X.X.XX............] duration: 3.413163149s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:02:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.413190249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X.X..............X..X......X................X.............] duration: 5.723757356s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:02:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.723788305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [................X.......X.....X............X.X.XX...........] duration: 3.48478748s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:02:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484832335s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X.X..............X..X......X................X............] duration: 5.892342541s trace: https://jaeger.dev.storj.io/trace/2f7262602a7ba484 FAIL (2025-12-09 05:02:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.892369061s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.................X.......X.....X............X.X.XX..........] duration: 3.538363299s trace: https://jaeger.dev.storj.io/trace/52b2f5efe2a1f132 FAIL (2025-12-09 05:02:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53839419s core uplink us-select-1 us1-placement-20

1
...
11636
11637
11638
11639
11640
...
17352