Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.914211591s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:29:57 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.914240195s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.448213299s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:29:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44823456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.437122065s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:29:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.437148267s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.950133753s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:29:15 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.950169921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.538451238s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:29:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538519169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.837832188s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:29:11 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.837862767s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.440782958s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:28:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.44080578s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.981677226s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:28:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.98172656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.88726364s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:28:30 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.887295492s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.399651142s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:28:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.399680378s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.521994161s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:28:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522026914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.087151466s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:27:47 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.087233131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.826020306s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:27:39 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.826051856s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.389118378s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:27:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.389147623s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.59428698s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:26:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.594318031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.122340015s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:26:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.12236892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.090964069s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:26:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.0909962s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.245434056s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:26:33 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.245469432s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.445243774s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:26:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.445261136s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.524153644s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:26:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524176657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.086092371s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:26:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.086120155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.129364138s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:25:50 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.129382362s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.415115954s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:25:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.415149938s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.431521943s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:25:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.431578441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.212522491s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:25:16 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.212566576s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.783045137s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:25:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.783072449s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.416599519s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:24:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.416657059s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.129049713s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:24:46 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.129069459s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.806381292s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:24:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.806410658s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.578156496s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:24:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578183137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.429678714s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:24:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.429718579s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.5205971s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:24:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520626927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.933709983s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:24:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.933738528s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.128056463s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:23:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.128086582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.424485127s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:23:48 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.42451815s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.432056952s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:23:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.432088221s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.437132996s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:23:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.43722937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.997820866s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:23:02 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.997894956s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.395609198s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:22:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.395638766s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.042888953s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:22:36 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.042919904s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.597938746s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:22:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.597968413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.410420133s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:22:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.410449729s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.971004707s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:21:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.97103151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.238893451s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:21:33 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.238917417s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.050822692s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:21:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.050867036s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.433812547s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:21:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.433845921s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.583244637s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:21:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.583281668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.239065705s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:20:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.239093648s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.478824975s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:20:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478860143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.413553948s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:20:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.413581301s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)