Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X.] duration: 3.461381946s trace: https://jaeger.dev.storj.io/trace/1d3731bd456a56c FAIL (2025-12-28 16:54:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461426019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..X.X...X] duration: 30.003400944s trace: https://jaeger.dev.storj.io/trace/635e74d6549d3335 FAIL (2025-12-28 16:54:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: KR6SNXFKK6EI5M7RG2ZU35PURWWZTKTEMYPL5I7PBYAS3VWDFBJQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 6MGBUS5XTP6XA2EQPBW5PNGA3QD5VDA36NYDPI6MSVMZYOXEMZCA) context deadline exceeded ======LOGS====== Dec 28 16:54:05.134696749 (553h25m5.005779784s) [upload=485199]: Starting upload Dec 28 16:54:05.134773735 (553h25m5.00585678s) [upload=485199]: Done starting upload Dec 28 16:54:05.134866843 (553h25m5.005949888s) [upload=485199]: Uploading segments... Dec 28 16:54:05.135003062 (553h25m5.006086107s) [upload=485199]: Got next segment. Inline: false Dec 28 16:54:05.135060711 (553h25m5.006143766s) [upload=485199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:54:05.135067645 (553h25m5.00615068s) [upload=485199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:54:05.378434988 (553h25m5.249518023s) [seg_pos={0 0}, upload=485199]: Begin upload segment... Dec 28 16:54:05.378488699 (553h25m5.249571734s) [seg_pos={0 0}, upload=485199]: Done begin upload segment. Dec 28 16:54:05.378522113 (553h25m5.249605158s) [noise=true , address=45.91.116.5:4298 , piece_id=6MGBUS5XTP6XA2EQPBW5PNGA3QD5VDA36NYDPI6MSVMZYOXEMZCA, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.378551279 (553h25m5.249634324s) [noise=true , address=145.79.227.13:4804 , piece_id=3NQDKB3DK7K3VISDW7G4NQ4SUNO3DIR4MFQPJVAKIUJLTKXPDBPA, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.379242295 (553h25m5.25032534s) [noise=true , address=45.91.116.10:4082 , piece_id=6XEECVQB56QPDF5HVWHXUGA5OZNI5BWBQ2YFWB3ZAIEJTJNJGU2A, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.379646204 (553h25m5.250729239s) [noise=true , address=145.79.227.32:4835 , piece_id=LIR6ZS2GBSHDOBW2NYFCW7OMVFIZXRMQ2XBE2VVBT5MTE6KEQPVQ, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.380049662 (553h25m5.251132707s) [noise=true , address=145.79.227.47:4267 , piece_id=JCHPTLOPRW6G2I42SMI2YCLCRPUNEI43JFSWXLF43ISVSSV25NIA, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.380454283 (553h25m5.251537318s) [noise=true , address=145.79.227.15:4876 , piece_id=MMTW7OKXHIZT75F4QUBLOIS4TQXDNR6E2WBNKW6RELBJPZQGWBJA, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.380848022 (553h25m5.251931067s) [noise=true , address=45.91.116.9:4263 , piece_id=KR6SNXFKK6EI5M7RG2ZU35PURWWZTKTEMYPL5I7PBYAS3VWDFBJQ, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.381281368 (553h25m5.252364413s) [noise=true , address=145.79.227.12:4765 , piece_id=ODL33E7YO6HKMWN6V5Y4D6YWV6CF3C3DWPVWBTMZDGIDNFMPOGJQ, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.383547232 (553h25m5.254630277s) [noise=true , address=145.79.227.46:4602 , piece_id=JSCK4NBJVVHF4V3A7PXGMFGVMBSZCQIDCOOMXZNCA2BEF7D6BE3A, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.385176113 (553h25m5.256259148s) [noise=true , address=77.90.31.16:4735 , piece_id=O7W6VA4RZMKVCY7SELP3UFU7RHUWXGDHLD3AS5FFOGBOWW2LQ3OQ, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.387185269 (553h25m5.258268304s) [noise=true , address=145.79.227.2:4114 , piece_id=ACSLUJIFYRD4GP4YEO6HQ3USGO743Y5KQD5JUUTFJA5UGO6T722A, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.389256975 (553h25m5.26034002s) [noise=true , address=45.91.116.8:4046 , piece_id=K6JRYCVXJNJDSZPAB627OM45VJQMNZLQKZEY7SLUDDTLGJPGBIKQ, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.391095374 (553h25m5.26217842s) [noise=true , address=145.79.227.17:4972 , piece_id=CB7SDBAYNJB6PLJDEJCWDAN66CX3YDXTVX4CA36OC6SCAXHWDJVQ, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.392026678 (553h25m5.263109713s) [noise=true , address=145.79.227.52:4447 , piece_id=67NRQPGIIZIHSAED5JPMKGNMC4UJE7NHRL6Y4KDBM4F55N2PNMYA, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.393943648 (553h25m5.265026683s) [noise=true , address=145.79.227.22:4509 , piece_id=XMXIJKHCM2XDQXNWGOV7OQ7PG4BVKMAGXLDKWUF6OS5NNGKNLV3Q, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:05.395111002 (553h25m5.266194047s) [noise=true , address=145.79.227.48:4303 , piece_id=NVO77LZO4KSEN6YP3RW5IVHEWE7W26Y2ND5SI2GBLLV6VJENUJ5A, seg_pos={0 0}, upload=485199]: Uploading piece... Dec 28 16:54:07.101409862 (553h25m6.972492907s) [upload=485199]: Next returned nil segment Dec 28 16:54:07.101424890 (553h25m6.972507935s) [upload=485199]: Waiting for error group managing segments... Dec 28 16:54:07.385200346 (553h25m7.256283391s) [noise=true , address=145.79.227.52:4447 , piece_id=67NRQPGIIZIHSAED5JPMKGNMC4UJE7NHRL6Y4KDBM4F55N2PNMYA, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.391111172 (553h25m7.262194217s) [noise=true , address=145.79.227.32:4835 , piece_id=LIR6ZS2GBSHDOBW2NYFCW7OMVFIZXRMQ2XBE2VVBT5MTE6KEQPVQ, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.391658253 (553h25m7.262741298s) [noise=true , address=145.79.227.46:4602 , piece_id=JSCK4NBJVVHF4V3A7PXGMFGVMBSZCQIDCOOMXZNCA2BEF7D6BE3A, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.392219582 (553h25m7.263302617s) [noise=true , address=145.79.227.15:4876 , piece_id=MMTW7OKXHIZT75F4QUBLOIS4TQXDNR6E2WBNKW6RELBJPZQGWBJA, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.396563674 (553h25m7.267646709s) [noise=true , address=145.79.227.2:4114 , piece_id=ACSLUJIFYRD4GP4YEO6HQ3USGO743Y5KQD5JUUTFJA5UGO6T722A, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.400388767 (553h25m7.271471802s) [noise=true , address=145.79.227.13:4804 , piece_id=3NQDKB3DK7K3VISDW7G4NQ4SUNO3DIR4MFQPJVAKIUJLTKXPDBPA, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.408987491 (553h25m7.280070556s) [noise=true , address=145.79.227.12:4765 , piece_id=ODL33E7YO6HKMWN6V5Y4D6YWV6CF3C3DWPVWBTMZDGIDNFMPOGJQ, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.412791314 (553h25m7.283874349s) [noise=true , address=145.79.227.48:4303 , piece_id=NVO77LZO4KSEN6YP3RW5IVHEWE7W26Y2ND5SI2GBLLV6VJENUJ5A, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.413269795 (553h25m7.28435284s) [noise=true , address=145.79.227.17:4972 , piece_id=CB7SDBAYNJB6PLJDEJCWDAN66CX3YDXTVX4CA36OC6SCAXHWDJVQ, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.416201237 (553h25m7.287284282s) [noise=true , address=145.79.227.47:4267 , piece_id=JCHPTLOPRW6G2I42SMI2YCLCRPUNEI43JFSWXLF43ISVSSV25NIA, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.419581585 (553h25m7.29066463s) [noise=true , address=145.79.227.22:4509 , piece_id=XMXIJKHCM2XDQXNWGOV7OQ7PG4BVKMAGXLDKWUF6OS5NNGKNLV3Q, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.635225367 (553h25m7.506308402s) [noise=true , address=45.91.116.9:4263 , piece_id=KR6SNXFKK6EI5M7RG2ZU35PURWWZTKTEMYPL5I7PBYAS3VWDFBJQ, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.659928185 (553h25m7.53101122s) [noise=true , address=45.91.116.5:4298 , piece_id=6MGBUS5XTP6XA2EQPBW5PNGA3QD5VDA36NYDPI6MSVMZYOXEMZCA, seg_pos={0 0}, upload=485199]: Done uploading piece. err: <nil> Dec 28 16:54:07.659946740 (553h25m7.531029785s) [seg_pos={0 0}, upload=485199]: Segment reached optimal threshold of 13 pieces. Dec 28 16:54:07.660137483 (553h25m7.531220518s) [noise=true , address=77.90.31.16:4735 , piece_id=O7W6VA4RZMKVCY7SELP3UFU7RHUWXGDHLD3AS5FFOGBOWW2LQ3OQ, seg_pos={0 0}, upload=485199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 16:54:07.660145890 (553h25m7.531228925s) [noise=true , address=77.90.31.16:4735 , piece_id=O7W6VA4RZMKVCY7SELP3UFU7RHUWXGDHLD3AS5FFOGBOWW2LQ3OQ, seg_pos={0 0}, upload=485199]: Long tail cancellation due to optimal threshold Dec 28 16:54:07.660406716 (553h25m7.531489751s) [noise=true , address=45.91.116.8:4046 , piece_id=K6JRYCVXJNJDSZPAB627OM45VJQMNZLQKZEY7SLUDDTLGJPGBIKQ, seg_pos={0 0}, upload=485199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 16:54:07.660413138 (553h25m7.531496173s) [noise=true , address=45.91.116.8:4046 , piece_id=K6JRYCVXJNJDSZPAB627OM45VJQMNZLQKZEY7SLUDDTLGJPGBIKQ, seg_pos={0 0}, upload=485199]: Long tail cancellation due to optimal threshold Dec 28 16:54:07.660529240 (553h25m7.531612275s) [noise=true , address=45.91.116.10:4082 , piece_id=6XEECVQB56QPDF5HVWHXUGA5OZNI5BWBQ2YFWB3ZAIEJTJNJGU2A, seg_pos={0 0}, upload=485199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:45744->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:45744->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45744->45.91.116.10:4082: use of closed network connection Dec 28 16:54:07.660534700 (553h25m7.531617735s) [noise=true , address=45.91.116.10:4082 , piece_id=6XEECVQB56QPDF5HVWHXUGA5OZNI5BWBQ2YFWB3ZAIEJTJNJGU2A, seg_pos={0 0}, upload=485199]: Long tail cancellation due to optimal threshold Dec 28 16:54:07.660594112 (553h25m7.531677158s) [seg_pos={0 0}, upload=485199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:54:07.660663034 (553h25m7.531746079s) [upload=485199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:54:07.660667682 (553h25m7.531750718s) [upload=485199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:54:07.946380679 (553h25m7.817463724s) [upload=485199]: Done uploading segments... Dec 28 16:54:07.946416878 (553h25m7.817499923s) [upload=485199]: Upload finished. err: <nil> Duration: 30.003417687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........................X.....................] duration: 34.573313798s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:53:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 34.573378992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.X] duration: 5.000933543s trace: https://jaeger.dev.storj.io/trace/1d3731bd456a56c FAIL (2025-12-28 16:53:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000957369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..X.X...] duration: 6.083678903s trace: https://jaeger.dev.storj.io/trace/2c58aed81269674e FAIL (2025-12-28 16:53:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA) context deadline exceeded ======LOGS====== Dec 28 16:50:31.014133804 (553h21m30.885216849s) [upload=485148]: Starting upload Dec 28 16:50:31.014188549 (553h21m30.885271594s) [upload=485148]: Done starting upload Dec 28 16:50:31.014316934 (553h21m30.885399969s) [upload=485148]: Uploading segments... Dec 28 16:50:31.014417646 (553h21m30.885500681s) [upload=485148]: Got next segment. Inline: false Dec 28 16:50:31.014453975 (553h21m30.88553701s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:50:31.014461739 (553h21m30.885544774s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:50:31.259425191 (553h21m31.130508226s) [seg_pos={0 0}, upload=485148]: Begin upload segment... Dec 28 16:50:31.259480116 (553h21m31.130563161s) [seg_pos={0 0}, upload=485148]: Done begin upload segment. Dec 28 16:50:31.259518339 (553h21m31.130601384s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.259667593 (553h21m31.130750638s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260237978 (553h21m31.131321023s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260989580 (553h21m31.132072625s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262223178 (553h21m31.133306224s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262714614 (553h21m31.133797659s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.263125897 (553h21m31.134208933s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264001054 (553h21m31.135084089s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264335561 (553h21m31.135418606s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265264420 (553h21m31.136347455s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265587545 (553h21m31.13667059s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266335730 (553h21m31.137418775s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266871220 (553h21m31.137954265s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266974436 (553h21m31.138057471s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.267865743 (553h21m31.138948778s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.268086885 (553h21m31.13916993s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:32.990126910 (553h21m32.861209955s) [upload=485148]: Next returned nil segment Dec 28 16:50:32.990138642 (553h21m32.861221677s) [upload=485148]: Waiting for error group managing segments... Dec 28 16:50:33.264434538 (553h21m33.135517574s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.269184994 (553h21m33.140268039s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.281408329 (553h21m33.152491364s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.282689799 (553h21m33.153772834s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.284906611 (553h21m33.155989656s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.288739920 (553h21m33.159822965s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.293547574 (553h21m33.164630619s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.303324111 (553h21m33.174407146s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.504756562 (553h21m33.375839668s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.510919187 (553h21m33.382002222s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.557039197 (553h21m33.428122232s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.564853036 (553h21m33.435936071s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650608739 (553h21m33.521691784s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650634618 (553h21m33.521717653s) [seg_pos={0 0}, upload=485148]: Segment reached optimal threshold of 13 pieces. Dec 28 16:50:33.650936643 (553h21m33.522019698s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.650966009 (553h21m33.522049044s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651062624 (553h21m33.522145679s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651081779 (553h21m33.522164815s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651246043 (553h21m33.522329078s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651253907 (553h21m33.522336952s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651274868 (553h21m33.522357913s) [seg_pos={0 0}, upload=485148]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:50:33.651398703 (553h21m33.522481748s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:50:33.651405957 (553h21m33.522489003s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:50:33.936242404 (553h21m33.807325449s) [upload=485148]: Done uploading segments... Dec 28 16:50:33.936265468 (553h21m33.807348503s) [upload=485148]: Upload finished. err: <nil> Duration: 6.083705235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........................X....................] duration: 27.834898061s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 27.834922818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX.] duration: 3.446670845s trace: https://jaeger.dev.storj.io/trace/5e3e4724a66559c8 FAIL (2025-12-28 16:52:35 UTC) orbiter warning: context deadline exceeded Duration: 3.446700451s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XXX] duration: 5.000732438s trace: https://jaeger.dev.storj.io/trace/5e3e4724a66559c8 FAIL (2025-12-28 16:52:16 UTC) orbiter warning: context deadline exceeded Duration: 5.000755372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..X.X..] duration: 5.814890196s trace: https://jaeger.dev.storj.io/trace/2c58aed81269674e FAIL (2025-12-28 16:52:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA) context deadline exceeded ======LOGS====== Dec 28 16:50:31.014133804 (553h21m30.885216849s) [upload=485148]: Starting upload Dec 28 16:50:31.014188549 (553h21m30.885271594s) [upload=485148]: Done starting upload Dec 28 16:50:31.014316934 (553h21m30.885399969s) [upload=485148]: Uploading segments... Dec 28 16:50:31.014417646 (553h21m30.885500681s) [upload=485148]: Got next segment. Inline: false Dec 28 16:50:31.014453975 (553h21m30.88553701s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:50:31.014461739 (553h21m30.885544774s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:50:31.259425191 (553h21m31.130508226s) [seg_pos={0 0}, upload=485148]: Begin upload segment... Dec 28 16:50:31.259480116 (553h21m31.130563161s) [seg_pos={0 0}, upload=485148]: Done begin upload segment. Dec 28 16:50:31.259518339 (553h21m31.130601384s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.259667593 (553h21m31.130750638s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260237978 (553h21m31.131321023s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260989580 (553h21m31.132072625s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262223178 (553h21m31.133306224s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262714614 (553h21m31.133797659s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.263125897 (553h21m31.134208933s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264001054 (553h21m31.135084089s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264335561 (553h21m31.135418606s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265264420 (553h21m31.136347455s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265587545 (553h21m31.13667059s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266335730 (553h21m31.137418775s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266871220 (553h21m31.137954265s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266974436 (553h21m31.138057471s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.267865743 (553h21m31.138948778s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.268086885 (553h21m31.13916993s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:32.990126910 (553h21m32.861209955s) [upload=485148]: Next returned nil segment Dec 28 16:50:32.990138642 (553h21m32.861221677s) [upload=485148]: Waiting for error group managing segments... Dec 28 16:50:33.264434538 (553h21m33.135517574s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.269184994 (553h21m33.140268039s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.281408329 (553h21m33.152491364s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.282689799 (553h21m33.153772834s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.284906611 (553h21m33.155989656s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.288739920 (553h21m33.159822965s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.293547574 (553h21m33.164630619s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.303324111 (553h21m33.174407146s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.504756562 (553h21m33.375839668s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.510919187 (553h21m33.382002222s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.557039197 (553h21m33.428122232s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.564853036 (553h21m33.435936071s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650608739 (553h21m33.521691784s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650634618 (553h21m33.521717653s) [seg_pos={0 0}, upload=485148]: Segment reached optimal threshold of 13 pieces. Dec 28 16:50:33.650936643 (553h21m33.522019698s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.650966009 (553h21m33.522049044s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651062624 (553h21m33.522145679s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651081779 (553h21m33.522164815s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651246043 (553h21m33.522329078s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651253907 (553h21m33.522336952s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651274868 (553h21m33.522357913s) [seg_pos={0 0}, upload=485148]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:50:33.651398703 (553h21m33.522481748s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:50:33.651405957 (553h21m33.522489003s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:50:33.936242404 (553h21m33.807325449s) [upload=485148]: Done uploading segments... Dec 28 16:50:33.936265468 (553h21m33.807348503s) [upload=485148]: Upload finished. err: <nil> Duration: 5.814927126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.............X.........................X...................] duration: 34.318545352s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:51:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 34.318572163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....XX] duration: 5.001673813s trace: https://jaeger.dev.storj.io/trace/ecf9325a57aeb1b FAIL (2025-12-28 16:51:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001692089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..X.X.] duration: 5.979099376s trace: https://jaeger.dev.storj.io/trace/2c58aed81269674e FAIL (2025-12-28 16:51:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA) context deadline exceeded ======LOGS====== Dec 28 16:50:31.014133804 (553h21m30.885216849s) [upload=485148]: Starting upload Dec 28 16:50:31.014188549 (553h21m30.885271594s) [upload=485148]: Done starting upload Dec 28 16:50:31.014316934 (553h21m30.885399969s) [upload=485148]: Uploading segments... Dec 28 16:50:31.014417646 (553h21m30.885500681s) [upload=485148]: Got next segment. Inline: false Dec 28 16:50:31.014453975 (553h21m30.88553701s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:50:31.014461739 (553h21m30.885544774s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:50:31.259425191 (553h21m31.130508226s) [seg_pos={0 0}, upload=485148]: Begin upload segment... Dec 28 16:50:31.259480116 (553h21m31.130563161s) [seg_pos={0 0}, upload=485148]: Done begin upload segment. Dec 28 16:50:31.259518339 (553h21m31.130601384s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.259667593 (553h21m31.130750638s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260237978 (553h21m31.131321023s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260989580 (553h21m31.132072625s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262223178 (553h21m31.133306224s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262714614 (553h21m31.133797659s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.263125897 (553h21m31.134208933s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264001054 (553h21m31.135084089s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264335561 (553h21m31.135418606s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265264420 (553h21m31.136347455s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265587545 (553h21m31.13667059s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266335730 (553h21m31.137418775s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266871220 (553h21m31.137954265s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266974436 (553h21m31.138057471s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.267865743 (553h21m31.138948778s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.268086885 (553h21m31.13916993s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:32.990126910 (553h21m32.861209955s) [upload=485148]: Next returned nil segment Dec 28 16:50:32.990138642 (553h21m32.861221677s) [upload=485148]: Waiting for error group managing segments... Dec 28 16:50:33.264434538 (553h21m33.135517574s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.269184994 (553h21m33.140268039s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.281408329 (553h21m33.152491364s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.282689799 (553h21m33.153772834s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.284906611 (553h21m33.155989656s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.288739920 (553h21m33.159822965s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.293547574 (553h21m33.164630619s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.303324111 (553h21m33.174407146s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.504756562 (553h21m33.375839668s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.510919187 (553h21m33.382002222s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.557039197 (553h21m33.428122232s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.564853036 (553h21m33.435936071s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650608739 (553h21m33.521691784s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650634618 (553h21m33.521717653s) [seg_pos={0 0}, upload=485148]: Segment reached optimal threshold of 13 pieces. Dec 28 16:50:33.650936643 (553h21m33.522019698s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.650966009 (553h21m33.522049044s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651062624 (553h21m33.522145679s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651081779 (553h21m33.522164815s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651246043 (553h21m33.522329078s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651253907 (553h21m33.522336952s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651274868 (553h21m33.522357913s) [seg_pos={0 0}, upload=485148]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:50:33.651398703 (553h21m33.522481748s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:50:33.651405957 (553h21m33.522489003s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:50:33.936242404 (553h21m33.807325449s) [upload=485148]: Done uploading segments... Dec 28 16:50:33.936265468 (553h21m33.807348503s) [upload=485148]: Upload finished. err: <nil> Duration: 5.979117862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.............X.........................X..................] duration: 15.65714293s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:51:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 15.6571699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..X.X] duration: 30.002870972s trace: https://jaeger.dev.storj.io/trace/2c58aed81269674e FAIL (2025-12-28 16:51:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA) context deadline exceeded ======LOGS====== Dec 28 16:50:31.014133804 (553h21m30.885216849s) [upload=485148]: Starting upload Dec 28 16:50:31.014188549 (553h21m30.885271594s) [upload=485148]: Done starting upload Dec 28 16:50:31.014316934 (553h21m30.885399969s) [upload=485148]: Uploading segments... Dec 28 16:50:31.014417646 (553h21m30.885500681s) [upload=485148]: Got next segment. Inline: false Dec 28 16:50:31.014453975 (553h21m30.88553701s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:50:31.014461739 (553h21m30.885544774s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:50:31.259425191 (553h21m31.130508226s) [seg_pos={0 0}, upload=485148]: Begin upload segment... Dec 28 16:50:31.259480116 (553h21m31.130563161s) [seg_pos={0 0}, upload=485148]: Done begin upload segment. Dec 28 16:50:31.259518339 (553h21m31.130601384s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.259667593 (553h21m31.130750638s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260237978 (553h21m31.131321023s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.260989580 (553h21m31.132072625s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262223178 (553h21m31.133306224s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.262714614 (553h21m31.133797659s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.263125897 (553h21m31.134208933s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264001054 (553h21m31.135084089s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.264335561 (553h21m31.135418606s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265264420 (553h21m31.136347455s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.265587545 (553h21m31.13667059s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266335730 (553h21m31.137418775s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266871220 (553h21m31.137954265s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.266974436 (553h21m31.138057471s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.267865743 (553h21m31.138948778s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:31.268086885 (553h21m31.13916993s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Uploading piece... Dec 28 16:50:32.990126910 (553h21m32.861209955s) [upload=485148]: Next returned nil segment Dec 28 16:50:32.990138642 (553h21m32.861221677s) [upload=485148]: Waiting for error group managing segments... Dec 28 16:50:33.264434538 (553h21m33.135517574s) [noise=true , address=145.79.227.7:4295 , piece_id=3RUJ464TVHHUZMXODJAIHNUU764UEKPKLIEU6ZVJVVQBKBCD75HA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.269184994 (553h21m33.140268039s) [noise=true , address=145.79.227.4:4185 , piece_id=W2KX4N5LHAUBICM2IXTTAJY7DWOBJKESGEERXHSBQBMXSJTEUVXA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.281408329 (553h21m33.152491364s) [noise=true , address=145.79.227.42:4121 , piece_id=5TQYCKOC5WY66JLAT4LUJAMHKPLANQT5LHAXHFWBLY57F723MY7Q, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.282689799 (553h21m33.153772834s) [noise=true , address=145.79.227.2:4114 , piece_id=UHOBSRNI4FH4QLL45KEA7QG7V7QAVKLCEM2Y3L5A2QBT6J62V5EA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.284906611 (553h21m33.155989656s) [noise=true , address=145.79.227.51:4409 , piece_id=U3WEIPQNJNFS37E6WE5YYIQSQRB4P3B3DPRHHHROWVVUZ3GN3BKA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.288739920 (553h21m33.159822965s) [noise=true , address=145.79.227.41:4086 , piece_id=G3YHPYHFZQOEU6KQ4MAA377AL54LOX5PNDJLBQEVGVYXLOG4K4LQ, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.293547574 (553h21m33.164630619s) [noise=true , address=145.79.227.17:4972 , piece_id=EZPSG2C7RXM5ACTYOV44EVOFQ52GAVKTTEZYPZBYCI4YCYNWLLJA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.303324111 (553h21m33.174407146s) [noise=true , address=145.79.227.13:4804 , piece_id=DYOKCHPWPNX3UKWW4DNIRL5YWEFWMISXGWTA3IWF5KMNWJNN2IPA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.504756562 (553h21m33.375839668s) [noise=true , address=45.91.116.14:4189 , piece_id=LBBQIAYN7UO65632W4OA43JX6VUM4QGTJZZB7CZAISRQGWL4F7KA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.510919187 (553h21m33.382002222s) [noise=true , address=45.91.116.6:4975 , piece_id=AWMOP36CEVUJLEAN2WKRBPOCB3RG7PUE6MAJ32HVBIPZ44QCOHWA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.557039197 (553h21m33.428122232s) [noise=true , address=45.91.116.9:4262 , piece_id=KEWIHFGYS6VHAANUTFXVNYK6UKRDAXGTZEUKHZD3L2YMH6VY7GYA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.564853036 (553h21m33.435936071s) [noise=true , address=45.91.116.11:4119 , piece_id=ECOUJBBWWNEEVLNNQDVQNQYPMLH2HWOG43Y57K26OG6OFGZZRIIA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650608739 (553h21m33.521691784s) [noise=true , address=77.90.31.21:4877 , piece_id=PPYJPYDTFRTRVK3E57RYG4LB4OPBTNFMXMFMCBVUZZO7GPQJG4ZA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: <nil> Dec 28 16:50:33.650634618 (553h21m33.521717653s) [seg_pos={0 0}, upload=485148]: Segment reached optimal threshold of 13 pieces. Dec 28 16:50:33.650936643 (553h21m33.522019698s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.650966009 (553h21m33.522049044s) [noise=true , address=45.91.116.10:4082 , piece_id=AGV7RQQCBMIRXQ7ITIYCAO5MAGGMCLZKRPMZPCXJ6KAXNRVHBRLA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651062624 (553h21m33.522145679s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651081779 (553h21m33.522164815s) [noise=true , address=77.90.31.16:4736 , piece_id=HJPOIZRXDCWRGSSGSHDQ6Q3WJ6DXQJ3PNXYBE776BCKHQXI5IDUA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651246043 (553h21m33.522329078s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 16:50:33.651253907 (553h21m33.522336952s) [noise=true , address=77.90.31.14:4664 , piece_id=WLNI3AM27KIEF47RSCAEC3O5TX7VODBCE5XO3OLZCA4XJUGONKCA, seg_pos={0 0}, upload=485148]: Long tail cancellation due to optimal threshold Dec 28 16:50:33.651274868 (553h21m33.522357913s) [seg_pos={0 0}, upload=485148]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:50:33.651398703 (553h21m33.522481748s) [upload=485148]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:50:33.651405957 (553h21m33.522489003s) [upload=485148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:50:33.936242404 (553h21m33.807325449s) [upload=485148]: Done uploading segments... Dec 28 16:50:33.936265468 (553h21m33.807348503s) [upload=485148]: Upload finished. err: <nil> Duration: 30.002891259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.............X.........................X.................] duration: 49.573282211s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:50:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 49.573300254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..X.] duration: 19.683384479s trace: https://jaeger.dev.storj.io/trace/193205ac093f6e1b FAIL (2025-12-28 16:50:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 16:48:47.442997588 (553h19m47.314080633s) [upload=485124]: Starting upload Dec 28 16:48:47.443057312 (553h19m47.314140357s) [upload=485124]: Done starting upload Dec 28 16:48:47.443261110 (553h19m47.314344145s) [upload=485124]: Uploading segments... Dec 28 16:48:47.443435623 (553h19m47.314518658s) [upload=485124]: Got next segment. Inline: false Dec 28 16:48:47.443497541 (553h19m47.314580586s) [upload=485124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:48:47.443504083 (553h19m47.314587118s) [upload=485124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:48:47.690038736 (553h19m47.561121791s) [seg_pos={0 0}, upload=485124]: Begin upload segment... Dec 28 16:48:47.690119600 (553h19m47.561202655s) [seg_pos={0 0}, upload=485124]: Done begin upload segment. Dec 28 16:48:47.690184534 (553h19m47.561267569s) [noise=true , address=45.91.116.4:4939 , piece_id=37ASIECSTEIRTOXSKWG4HT22BMA3X2546JPISRZ22LH5NXUPMWPQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.690311316 (553h19m47.561394351s) [noise=true , address=145.79.227.51:4411 , piece_id=QPTSU3SODCBTMT7G3NXCE4SZHVGYISVHROROQ6XQQDPOFWXQIGQQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.690882102 (553h19m47.561965147s) [noise=true , address=145.79.227.39:4015 , piece_id=7TEX5F5AHCXY6HF267GMGRVB7RU4ZBBOLRWG5XFLB3KOIZNFXZYQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691010407 (553h19m47.562093452s) [noise=true , address=45.91.116.10:4081 , piece_id=XKRED6ZKQY5QLJQNPZ4PQ3KK2NM6DGD65JX2OANCTLTVUIKK6E2Q, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691281744 (553h19m47.562364779s) [noise=true , address=45.91.116.14:4189 , piece_id=QD77QVJWDSXH2DKVBAL7RBR3TZP4VMIONKBQEU6YAEIYTQLVVO3Q, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691452699 (553h19m47.562535744s) [noise=true , address=45.91.116.7:4009 , piece_id=3DNT44EV5R7ISQ2CB5FIFXJZEAQQN2HGDSJW2ASJN6QMX53KT2UQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691672377 (553h19m47.562755452s) [noise=true , address=145.79.227.8:4330 , piece_id=XANBF65W66LBQ4GFJW77Q5L42LYVEHTJR3RX3XTOF7XAGLHPNBHQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691787957 (553h19m47.562870992s) [noise=true , address=145.79.227.42:4122 , piece_id=B2IJKFT6GTGYKKHH22UDU66MTGMHY4OX3V2UVLCEMNQ4HSLC3MKA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692056188 (553h19m47.563139223s) [noise=true , address=77.90.31.12:4589 , piece_id=IDVZC4WU67RR2PZ2YVP4RE5HTU5ZWA5YMW7QB5P3EYDBEOM6B3IQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692117876 (553h19m47.563200921s) [noise=true , address=145.79.227.19:4079 , piece_id=3NH3QUY5SI634ZVLQ55WHMU4U77H4CCWOPFC44HXA35VLQLVY4PQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692343835 (553h19m47.563426871s) [noise=true , address=145.79.227.38:4978 , piece_id=LQJVBKIYI3FTABIKVDYXSAU4DBGYTJDPHFCCDB3WWWYN6QGNJ3KA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692462923 (553h19m47.563545968s) [noise=true , address=145.79.227.5:4223 , piece_id=YN73CQ5JGBOHO6ZE44MONFWCJCTU6A67WQHUXAEDKOLA2JPWJMKA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692793452 (553h19m47.563876487s) [noise=true , address=145.79.227.3:4150 , piece_id=V5XGZNRDC3YBXWOFYNRUHEUDTW4HQESMJLCQT3A4R6YMJELXFRBQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.693118460 (553h19m47.564201495s) [noise=true , address=145.79.227.17:4972 , piece_id=HF4YLJG54JIUI3KYWYNDNA3FK7HOPA6K363IQMDFBCWEQJODAKWA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.693562486 (553h19m47.564645521s) [noise=true , address=145.79.227.37:4566 , piece_id=JSK65KMD7RBXQ754KEILTOFMVRW6ZGJYZEJOFJM4D5QPH62QZXFA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.694003075 (553h19m47.56508611s) [noise=true , address=45.91.116.5:4300 , piece_id=WXW7SHIU3YSOTT5AOX3FQUMKCCZSEF7MRGEYL4UC5EDMOAXEZLJQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:49.399870575 (553h19m49.270953611s) [upload=485124]: Next returned nil segment Dec 28 16:48:49.399883860 (553h19m49.270966895s) [upload=485124]: Waiting for error group managing segments... Dec 28 16:48:49.681350511 (553h19m49.552433546s) [noise=true , address=145.79.227.51:4411 , piece_id=QPTSU3SODCBTMT7G3NXCE4SZHVGYISVHROROQ6XQQDPOFWXQIGQQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.684895742 (553h19m49.555978777s) [noise=true , address=145.79.227.5:4223 , piece_id=YN73CQ5JGBOHO6ZE44MONFWCJCTU6A67WQHUXAEDKOLA2JPWJMKA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.697016592 (553h19m49.568099637s) [noise=true , address=145.79.227.19:4079 , piece_id=3NH3QUY5SI634ZVLQ55WHMU4U77H4CCWOPFC44HXA35VLQLVY4PQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.699985174 (553h19m49.571068219s) [noise=true , address=145.79.227.39:4015 , piece_id=7TEX5F5AHCXY6HF267GMGRVB7RU4ZBBOLRWG5XFLB3KOIZNFXZYQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.701135825 (553h19m49.57221886s) [noise=true , address=145.79.227.17:4972 , piece_id=HF4YLJG54JIUI3KYWYNDNA3FK7HOPA6K363IQMDFBCWEQJODAKWA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.707349807 (553h19m49.578432842s) [noise=true , address=145.79.227.38:4978 , piece_id=LQJVBKIYI3FTABIKVDYXSAU4DBGYTJDPHFCCDB3WWWYN6QGNJ3KA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.708050662 (553h19m49.579133707s) [noise=true , address=145.79.227.42:4122 , piece_id=B2IJKFT6GTGYKKHH22UDU66MTGMHY4OX3V2UVLCEMNQ4HSLC3MKA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.718109345 (553h19m49.58919238s) [noise=true , address=145.79.227.8:4330 , piece_id=XANBF65W66LBQ4GFJW77Q5L42LYVEHTJR3RX3XTOF7XAGLHPNBHQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.728024435 (553h19m49.59910747s) [noise=true , address=145.79.227.37:4566 , piece_id=JSK65KMD7RBXQ754KEILTOFMVRW6ZGJYZEJOFJM4D5QPH62QZXFA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.728684382 (553h19m49.599767427s) [noise=true , address=145.79.227.3:4150 , piece_id=V5XGZNRDC3YBXWOFYNRUHEUDTW4HQESMJLCQT3A4R6YMJELXFRBQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.936163117 (553h19m49.807246162s) [noise=true , address=45.91.116.7:4009 , piece_id=3DNT44EV5R7ISQ2CB5FIFXJZEAQQN2HGDSJW2ASJN6QMX53KT2UQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.998719164 (553h19m49.869802209s) [noise=true , address=77.90.31.12:4589 , piece_id=IDVZC4WU67RR2PZ2YVP4RE5HTU5ZWA5YMW7QB5P3EYDBEOM6B3IQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:50.025040335 (553h19m49.89612337s) [noise=true , address=45.91.116.10:4081 , piece_id=XKRED6ZKQY5QLJQNPZ4PQ3KK2NM6DGD65JX2OANCTLTVUIKK6E2Q, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:50.025059340 (553h19m49.896142375s) [seg_pos={0 0}, upload=485124]: Segment reached optimal threshold of 13 pieces. Dec 28 16:48:50.025247399 (553h19m49.896330444s) [noise=true , address=45.91.116.14:4189 , piece_id=QD77QVJWDSXH2DKVBAL7RBR3TZP4VMIONKBQEU6YAEIYTQLVVO3Q, seg_pos={0 0}, upload=485124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 16:48:50.025253891 (553h19m49.896336936s) [noise=true , address=45.91.116.14:4189 , piece_id=QD77QVJWDSXH2DKVBAL7RBR3TZP4VMIONKBQEU6YAEIYTQLVVO3Q, seg_pos={0 0}, upload=485124]: Long tail cancellation due to optimal threshold Dec 28 16:48:50.025570925 (553h19m49.89665396s) [noise=true , address=45.91.116.5:4300 , piece_id=WXW7SHIU3YSOTT5AOX3FQUMKCCZSEF7MRGEYL4UC5EDMOAXEZLJQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 16:48:50.025578178 (553h19m49.896661213s) [noise=true , address=45.91.116.5:4300 , piece_id=WXW7SHIU3YSOTT5AOX3FQUMKCCZSEF7MRGEYL4UC5EDMOAXEZLJQ, seg_pos={0 0}, upload=485124]: Long tail cancellation due to optimal threshold Dec 28 16:48:50.025914008 (553h19m49.896997053s) [noise=true , address=45.91.116.4:4939 , piece_id=37ASIECSTEIRTOXSKWG4HT22BMA3X2546JPISRZ22LH5NXUPMWPQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55062->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55062->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55062->45.91.116.4:4939: use of closed network connection Dec 28 16:48:50.025923055 (553h19m49.8970061s) [noise=true , address=45.91.116.4:4939 , piece_id=37ASIECSTEIRTOXSKWG4HT22BMA3X2546JPISRZ22LH5NXUPMWPQ, seg_pos={0 0}, upload=485124]: Long tail cancellation due to optimal threshold Dec 28 16:48:50.025945959 (553h19m49.897028994s) [seg_pos={0 0}, upload=485124]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:48:50.026078150 (553h19m49.897161195s) [upload=485124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:48:50.026084403 (553h19m49.897167458s) [upload=485124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:48:50.333039697 (553h19m50.204122772s) [upload=485124]: Done uploading segments... Dec 28 16:48:50.333074272 (553h19m50.204157358s) [upload=485124]: Upload finished. err: <nil> Duration: 19.683425809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [.XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....X] duration: 5.000796111s trace: https://jaeger.dev.storj.io/trace/166b62633eb584d2 FAIL (2025-12-28 16:49:59 UTC) orbiter warning: context deadline exceeded Duration: 5.000822061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.............X.........................X................] duration: 33.198773054s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:49:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 33.198802148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..X] duration: 30.004243948s trace: https://jaeger.dev.storj.io/trace/193205ac093f6e1b FAIL (2025-12-28 16:49:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 16:48:47.442997588 (553h19m47.314080633s) [upload=485124]: Starting upload Dec 28 16:48:47.443057312 (553h19m47.314140357s) [upload=485124]: Done starting upload Dec 28 16:48:47.443261110 (553h19m47.314344145s) [upload=485124]: Uploading segments... Dec 28 16:48:47.443435623 (553h19m47.314518658s) [upload=485124]: Got next segment. Inline: false Dec 28 16:48:47.443497541 (553h19m47.314580586s) [upload=485124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:48:47.443504083 (553h19m47.314587118s) [upload=485124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:48:47.690038736 (553h19m47.561121791s) [seg_pos={0 0}, upload=485124]: Begin upload segment... Dec 28 16:48:47.690119600 (553h19m47.561202655s) [seg_pos={0 0}, upload=485124]: Done begin upload segment. Dec 28 16:48:47.690184534 (553h19m47.561267569s) [noise=true , address=45.91.116.4:4939 , piece_id=37ASIECSTEIRTOXSKWG4HT22BMA3X2546JPISRZ22LH5NXUPMWPQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.690311316 (553h19m47.561394351s) [noise=true , address=145.79.227.51:4411 , piece_id=QPTSU3SODCBTMT7G3NXCE4SZHVGYISVHROROQ6XQQDPOFWXQIGQQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.690882102 (553h19m47.561965147s) [noise=true , address=145.79.227.39:4015 , piece_id=7TEX5F5AHCXY6HF267GMGRVB7RU4ZBBOLRWG5XFLB3KOIZNFXZYQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691010407 (553h19m47.562093452s) [noise=true , address=45.91.116.10:4081 , piece_id=XKRED6ZKQY5QLJQNPZ4PQ3KK2NM6DGD65JX2OANCTLTVUIKK6E2Q, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691281744 (553h19m47.562364779s) [noise=true , address=45.91.116.14:4189 , piece_id=QD77QVJWDSXH2DKVBAL7RBR3TZP4VMIONKBQEU6YAEIYTQLVVO3Q, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691452699 (553h19m47.562535744s) [noise=true , address=45.91.116.7:4009 , piece_id=3DNT44EV5R7ISQ2CB5FIFXJZEAQQN2HGDSJW2ASJN6QMX53KT2UQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691672377 (553h19m47.562755452s) [noise=true , address=145.79.227.8:4330 , piece_id=XANBF65W66LBQ4GFJW77Q5L42LYVEHTJR3RX3XTOF7XAGLHPNBHQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.691787957 (553h19m47.562870992s) [noise=true , address=145.79.227.42:4122 , piece_id=B2IJKFT6GTGYKKHH22UDU66MTGMHY4OX3V2UVLCEMNQ4HSLC3MKA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692056188 (553h19m47.563139223s) [noise=true , address=77.90.31.12:4589 , piece_id=IDVZC4WU67RR2PZ2YVP4RE5HTU5ZWA5YMW7QB5P3EYDBEOM6B3IQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692117876 (553h19m47.563200921s) [noise=true , address=145.79.227.19:4079 , piece_id=3NH3QUY5SI634ZVLQ55WHMU4U77H4CCWOPFC44HXA35VLQLVY4PQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692343835 (553h19m47.563426871s) [noise=true , address=145.79.227.38:4978 , piece_id=LQJVBKIYI3FTABIKVDYXSAU4DBGYTJDPHFCCDB3WWWYN6QGNJ3KA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692462923 (553h19m47.563545968s) [noise=true , address=145.79.227.5:4223 , piece_id=YN73CQ5JGBOHO6ZE44MONFWCJCTU6A67WQHUXAEDKOLA2JPWJMKA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.692793452 (553h19m47.563876487s) [noise=true , address=145.79.227.3:4150 , piece_id=V5XGZNRDC3YBXWOFYNRUHEUDTW4HQESMJLCQT3A4R6YMJELXFRBQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.693118460 (553h19m47.564201495s) [noise=true , address=145.79.227.17:4972 , piece_id=HF4YLJG54JIUI3KYWYNDNA3FK7HOPA6K363IQMDFBCWEQJODAKWA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.693562486 (553h19m47.564645521s) [noise=true , address=145.79.227.37:4566 , piece_id=JSK65KMD7RBXQ754KEILTOFMVRW6ZGJYZEJOFJM4D5QPH62QZXFA, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:47.694003075 (553h19m47.56508611s) [noise=true , address=45.91.116.5:4300 , piece_id=WXW7SHIU3YSOTT5AOX3FQUMKCCZSEF7MRGEYL4UC5EDMOAXEZLJQ, seg_pos={0 0}, upload=485124]: Uploading piece... Dec 28 16:48:49.399870575 (553h19m49.270953611s) [upload=485124]: Next returned nil segment Dec 28 16:48:49.399883860 (553h19m49.270966895s) [upload=485124]: Waiting for error group managing segments... Dec 28 16:48:49.681350511 (553h19m49.552433546s) [noise=true , address=145.79.227.51:4411 , piece_id=QPTSU3SODCBTMT7G3NXCE4SZHVGYISVHROROQ6XQQDPOFWXQIGQQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.684895742 (553h19m49.555978777s) [noise=true , address=145.79.227.5:4223 , piece_id=YN73CQ5JGBOHO6ZE44MONFWCJCTU6A67WQHUXAEDKOLA2JPWJMKA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.697016592 (553h19m49.568099637s) [noise=true , address=145.79.227.19:4079 , piece_id=3NH3QUY5SI634ZVLQ55WHMU4U77H4CCWOPFC44HXA35VLQLVY4PQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.699985174 (553h19m49.571068219s) [noise=true , address=145.79.227.39:4015 , piece_id=7TEX5F5AHCXY6HF267GMGRVB7RU4ZBBOLRWG5XFLB3KOIZNFXZYQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.701135825 (553h19m49.57221886s) [noise=true , address=145.79.227.17:4972 , piece_id=HF4YLJG54JIUI3KYWYNDNA3FK7HOPA6K363IQMDFBCWEQJODAKWA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.707349807 (553h19m49.578432842s) [noise=true , address=145.79.227.38:4978 , piece_id=LQJVBKIYI3FTABIKVDYXSAU4DBGYTJDPHFCCDB3WWWYN6QGNJ3KA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.708050662 (553h19m49.579133707s) [noise=true , address=145.79.227.42:4122 , piece_id=B2IJKFT6GTGYKKHH22UDU66MTGMHY4OX3V2UVLCEMNQ4HSLC3MKA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.718109345 (553h19m49.58919238s) [noise=true , address=145.79.227.8:4330 , piece_id=XANBF65W66LBQ4GFJW77Q5L42LYVEHTJR3RX3XTOF7XAGLHPNBHQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.728024435 (553h19m49.59910747s) [noise=true , address=145.79.227.37:4566 , piece_id=JSK65KMD7RBXQ754KEILTOFMVRW6ZGJYZEJOFJM4D5QPH62QZXFA, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.728684382 (553h19m49.599767427s) [noise=true , address=145.79.227.3:4150 , piece_id=V5XGZNRDC3YBXWOFYNRUHEUDTW4HQESMJLCQT3A4R6YMJELXFRBQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.936163117 (553h19m49.807246162s) [noise=true , address=45.91.116.7:4009 , piece_id=3DNT44EV5R7ISQ2CB5FIFXJZEAQQN2HGDSJW2ASJN6QMX53KT2UQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:49.998719164 (553h19m49.869802209s) [noise=true , address=77.90.31.12:4589 , piece_id=IDVZC4WU67RR2PZ2YVP4RE5HTU5ZWA5YMW7QB5P3EYDBEOM6B3IQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:50.025040335 (553h19m49.89612337s) [noise=true , address=45.91.116.10:4081 , piece_id=XKRED6ZKQY5QLJQNPZ4PQ3KK2NM6DGD65JX2OANCTLTVUIKK6E2Q, seg_pos={0 0}, upload=485124]: Done uploading piece. err: <nil> Dec 28 16:48:50.025059340 (553h19m49.896142375s) [seg_pos={0 0}, upload=485124]: Segment reached optimal threshold of 13 pieces. Dec 28 16:48:50.025247399 (553h19m49.896330444s) [noise=true , address=45.91.116.14:4189 , piece_id=QD77QVJWDSXH2DKVBAL7RBR3TZP4VMIONKBQEU6YAEIYTQLVVO3Q, seg_pos={0 0}, upload=485124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 16:48:50.025253891 (553h19m49.896336936s) [noise=true , address=45.91.116.14:4189 , piece_id=QD77QVJWDSXH2DKVBAL7RBR3TZP4VMIONKBQEU6YAEIYTQLVVO3Q, seg_pos={0 0}, upload=485124]: Long tail cancellation due to optimal threshold Dec 28 16:48:50.025570925 (553h19m49.89665396s) [noise=true , address=45.91.116.5:4300 , piece_id=WXW7SHIU3YSOTT5AOX3FQUMKCCZSEF7MRGEYL4UC5EDMOAXEZLJQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 16:48:50.025578178 (553h19m49.896661213s) [noise=true , address=45.91.116.5:4300 , piece_id=WXW7SHIU3YSOTT5AOX3FQUMKCCZSEF7MRGEYL4UC5EDMOAXEZLJQ, seg_pos={0 0}, upload=485124]: Long tail cancellation due to optimal threshold Dec 28 16:48:50.025914008 (553h19m49.896997053s) [noise=true , address=45.91.116.4:4939 , piece_id=37ASIECSTEIRTOXSKWG4HT22BMA3X2546JPISRZ22LH5NXUPMWPQ, seg_pos={0 0}, upload=485124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55062->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55062->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55062->45.91.116.4:4939: use of closed network connection Dec 28 16:48:50.025923055 (553h19m49.8970061s) [noise=true , address=45.91.116.4:4939 , piece_id=37ASIECSTEIRTOXSKWG4HT22BMA3X2546JPISRZ22LH5NXUPMWPQ, seg_pos={0 0}, upload=485124]: Long tail cancellation due to optimal threshold Dec 28 16:48:50.025945959 (553h19m49.897028994s) [seg_pos={0 0}, upload=485124]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:48:50.026078150 (553h19m49.897161195s) [upload=485124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:48:50.026084403 (553h19m49.897167458s) [upload=485124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:48:50.333039697 (553h19m50.204122772s) [upload=485124]: Done uploading segments... Dec 28 16:48:50.333074272 (553h19m50.204157358s) [upload=485124]: Upload finished. err: <nil> Duration: 30.004267092s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.....] duration: 3.673942776s trace: https://jaeger.dev.storj.io/trace/1ab9abc21447e609 FAIL (2025-12-28 16:49:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.673961031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............X.........................X...............] duration: 21.331957156s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:48:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 21.331987974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X....] duration: 4.312436469s trace: https://jaeger.dev.storj.io/trace/1ab9abc21447e609 FAIL (2025-12-28 16:48:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.312464542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX..] duration: 29.740613907s trace: https://jaeger.dev.storj.io/trace/3d51b27ec0a0f509 FAIL (2025-12-28 16:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA) context deadline exceeded ======LOGS====== Dec 28 16:45:45.080572784 (553h16m44.951655829s) [upload=485084]: Starting upload Dec 28 16:45:45.080625815 (553h16m44.95170885s) [upload=485084]: Done starting upload Dec 28 16:45:45.080832970 (553h16m44.951916005s) [upload=485084]: Uploading segments... Dec 28 16:45:45.080876843 (553h16m44.951959878s) [upload=485084]: Got next segment. Inline: false Dec 28 16:45:45.080914545 (553h16m44.95199758s) [upload=485084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:45:45.080918873 (553h16m44.952001908s) [upload=485084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:45:45.337769280 (553h16m45.208852315s) [seg_pos={0 0}, upload=485084]: Begin upload segment... Dec 28 16:45:45.337924525 (553h16m45.20900757s) [seg_pos={0 0}, upload=485084]: Done begin upload segment. Dec 28 16:45:45.338005900 (553h16m45.209088935s) [noise=true , address=145.79.227.3:4151 , piece_id=TQI7V76ULX6YLDBDWNNCTXHTWTF3P2SYGLHPN5635M52VLZUJRSQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338024616 (553h16m45.209107661s) [noise=true , address=145.79.227.19:4078 , piece_id=VQMJIONT6M3YTEXBQ6KNILMFG7SS7WNZ2TTHZSMS7TGU47U33C2A, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338393178 (553h16m45.209476213s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338653052 (553h16m45.209736087s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338915873 (553h16m45.209998908s) [noise=true , address=145.79.227.51:4410 , piece_id=JZ4V6ID5WY35OK25C5SWTC7LCMY3AZ6J4LNLADFFWUKUITM67FDQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339188031 (553h16m45.210271066s) [noise=true , address=145.79.227.7:4295 , piece_id=TV4LICWPQ4RL4XFZ55ZET6OLVADDZDKFS22NTPBCSHGPKV6S6DIQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339195806 (553h16m45.210278841s) [noise=true , address=77.90.31.10:4519 , piece_id=7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339459137 (553h16m45.210542182s) [noise=true , address=145.79.227.38:4979 , piece_id=MNO6BEYQ7Y2Y4CYQ63QV4RRU5W7SMDK2LGG6JRUZ6IV45NK3QMOQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339494494 (553h16m45.21057753s) [noise=true , address=145.79.227.27:4690 , piece_id=6ZHH6H7ULO3A5WK7USEYM3RJFSKZDBXHI7YUZYFMXVRM2P5YJVIQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339745392 (553h16m45.210828437s) [noise=true , address=145.79.227.10:4401 , piece_id=R5R4MZQSPATWYWNLRGGUCMCUSEMHLVQQGOJWBPI3DQYSSUZPANSQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339769458 (553h16m45.210852503s) [noise=true , address=145.79.227.21:4473 , piece_id=7EXSPSXDE5Q2DD2KMZFB765BYKUBUNI4A6C5KDJLQ7SEXPK4ONCQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340019815 (553h16m45.21110285s) [noise=true , address=145.79.227.32:4836 , piece_id=2MDYMLYUAVLK3HK4KWWJWFOE6LG5KLAIT7QCZNP762GXLNW36DRQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340038290 (553h16m45.211121335s) [noise=true , address=145.79.227.14:4839 , piece_id=BSK4K6LL7EW2QATY54QTMUEIWUQ6VPWFMPEW3UPQS72QPLL6CMYQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340294728 (553h16m45.211377773s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340324325 (553h16m45.21140737s) [noise=true , address=145.79.227.42:4121 , piece_id=DAQ6KEYTIWDER4GAV6GFVHFZ5KEVUXTGSM2S2Q25GRZTJFTC3NWA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.341623909 (553h16m45.212706944s) [noise=true , address=45.91.116.8:4046 , piece_id=XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:47.048374882 (553h16m46.919457917s) [upload=485084]: Next returned nil segment Dec 28 16:45:47.048386494 (553h16m46.919469529s) [upload=485084]: Waiting for error group managing segments... Dec 28 16:45:47.334974136 (553h16m47.206057171s) [noise=true , address=145.79.227.19:4078 , piece_id=VQMJIONT6M3YTEXBQ6KNILMFG7SS7WNZ2TTHZSMS7TGU47U33C2A, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.335852137 (553h16m47.206935182s) [noise=true , address=145.79.227.32:4836 , piece_id=2MDYMLYUAVLK3HK4KWWJWFOE6LG5KLAIT7QCZNP762GXLNW36DRQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.344227265 (553h16m47.2153103s) [noise=true , address=145.79.227.10:4401 , piece_id=R5R4MZQSPATWYWNLRGGUCMCUSEMHLVQQGOJWBPI3DQYSSUZPANSQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.346648114 (553h16m47.21773115s) [noise=true , address=145.79.227.7:4295 , piece_id=TV4LICWPQ4RL4XFZ55ZET6OLVADDZDKFS22NTPBCSHGPKV6S6DIQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.347586722 (553h16m47.218669767s) [noise=true , address=145.79.227.3:4151 , piece_id=TQI7V76ULX6YLDBDWNNCTXHTWTF3P2SYGLHPN5635M52VLZUJRSQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.348580264 (553h16m47.219663309s) [noise=true , address=145.79.227.27:4690 , piece_id=6ZHH6H7ULO3A5WK7USEYM3RJFSKZDBXHI7YUZYFMXVRM2P5YJVIQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.364151624 (553h16m47.235234709s) [noise=true , address=145.79.227.51:4410 , piece_id=JZ4V6ID5WY35OK25C5SWTC7LCMY3AZ6J4LNLADFFWUKUITM67FDQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.365004788 (553h16m47.236087823s) [noise=true , address=145.79.227.21:4473 , piece_id=7EXSPSXDE5Q2DD2KMZFB765BYKUBUNI4A6C5KDJLQ7SEXPK4ONCQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.366190105 (553h16m47.23727314s) [noise=true , address=145.79.227.38:4979 , piece_id=MNO6BEYQ7Y2Y4CYQ63QV4RRU5W7SMDK2LGG6JRUZ6IV45NK3QMOQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.368810305 (553h16m47.239893361s) [noise=true , address=145.79.227.42:4121 , piece_id=DAQ6KEYTIWDER4GAV6GFVHFZ5KEVUXTGSM2S2Q25GRZTJFTC3NWA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.512574536 (553h16m47.383657582s) [noise=true , address=77.90.31.10:4519 , piece_id=7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.536220132 (553h16m47.407303167s) [noise=true , address=145.79.227.14:4839 , piece_id=BSK4K6LL7EW2QATY54QTMUEIWUQ6VPWFMPEW3UPQS72QPLL6CMYQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.564997609 (553h16m47.436080654s) [noise=true , address=45.91.116.8:4046 , piece_id=XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.565016785 (553h16m47.43609982s) [seg_pos={0 0}, upload=485084]: Segment reached optimal threshold of 13 pieces. Dec 28 16:45:47.565204964 (553h16m47.436287999s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection Dec 28 16:45:47.565211506 (553h16m47.436294541s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565257914 (553h16m47.436340949s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 16:45:47.565265890 (553h16m47.436348925s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565418040 (553h16m47.436501085s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 16:45:47.565424452 (553h16m47.436507487s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565438438 (553h16m47.436521473s) [seg_pos={0 0}, upload=485084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:45:47.565506668 (553h16m47.436589703s) [upload=485084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:45:47.565511137 (553h16m47.436594182s) [upload=485084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:45:47.854627794 (553h16m47.725710839s) [upload=485084]: Done uploading segments... Dec 28 16:45:47.854653423 (553h16m47.725736468s) [upload=485084]: Upload finished. err: <nil> Duration: 29.740640688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............X.........................X..............] duration: 29.394205737s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 29.394246735s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X...] duration: 5.001496274s trace: https://jaeger.dev.storj.io/trace/1ab9abc21447e609 FAIL (2025-12-28 16:47:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001514659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X..] duration: 3.456988279s trace: https://jaeger.dev.storj.io/trace/1ab9abc21447e609 FAIL (2025-12-28 16:47:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457028537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............X.........................X.............] duration: 43.615204173s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:47:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 43.615245221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX.] duration: 5.995422168s trace: https://jaeger.dev.storj.io/trace/3d51b27ec0a0f509 FAIL (2025-12-28 16:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA) context deadline exceeded ======LOGS====== Dec 28 16:45:45.080572784 (553h16m44.951655829s) [upload=485084]: Starting upload Dec 28 16:45:45.080625815 (553h16m44.95170885s) [upload=485084]: Done starting upload Dec 28 16:45:45.080832970 (553h16m44.951916005s) [upload=485084]: Uploading segments... Dec 28 16:45:45.080876843 (553h16m44.951959878s) [upload=485084]: Got next segment. Inline: false Dec 28 16:45:45.080914545 (553h16m44.95199758s) [upload=485084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:45:45.080918873 (553h16m44.952001908s) [upload=485084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:45:45.337769280 (553h16m45.208852315s) [seg_pos={0 0}, upload=485084]: Begin upload segment... Dec 28 16:45:45.337924525 (553h16m45.20900757s) [seg_pos={0 0}, upload=485084]: Done begin upload segment. Dec 28 16:45:45.338005900 (553h16m45.209088935s) [noise=true , address=145.79.227.3:4151 , piece_id=TQI7V76ULX6YLDBDWNNCTXHTWTF3P2SYGLHPN5635M52VLZUJRSQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338024616 (553h16m45.209107661s) [noise=true , address=145.79.227.19:4078 , piece_id=VQMJIONT6M3YTEXBQ6KNILMFG7SS7WNZ2TTHZSMS7TGU47U33C2A, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338393178 (553h16m45.209476213s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338653052 (553h16m45.209736087s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338915873 (553h16m45.209998908s) [noise=true , address=145.79.227.51:4410 , piece_id=JZ4V6ID5WY35OK25C5SWTC7LCMY3AZ6J4LNLADFFWUKUITM67FDQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339188031 (553h16m45.210271066s) [noise=true , address=145.79.227.7:4295 , piece_id=TV4LICWPQ4RL4XFZ55ZET6OLVADDZDKFS22NTPBCSHGPKV6S6DIQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339195806 (553h16m45.210278841s) [noise=true , address=77.90.31.10:4519 , piece_id=7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339459137 (553h16m45.210542182s) [noise=true , address=145.79.227.38:4979 , piece_id=MNO6BEYQ7Y2Y4CYQ63QV4RRU5W7SMDK2LGG6JRUZ6IV45NK3QMOQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339494494 (553h16m45.21057753s) [noise=true , address=145.79.227.27:4690 , piece_id=6ZHH6H7ULO3A5WK7USEYM3RJFSKZDBXHI7YUZYFMXVRM2P5YJVIQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339745392 (553h16m45.210828437s) [noise=true , address=145.79.227.10:4401 , piece_id=R5R4MZQSPATWYWNLRGGUCMCUSEMHLVQQGOJWBPI3DQYSSUZPANSQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339769458 (553h16m45.210852503s) [noise=true , address=145.79.227.21:4473 , piece_id=7EXSPSXDE5Q2DD2KMZFB765BYKUBUNI4A6C5KDJLQ7SEXPK4ONCQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340019815 (553h16m45.21110285s) [noise=true , address=145.79.227.32:4836 , piece_id=2MDYMLYUAVLK3HK4KWWJWFOE6LG5KLAIT7QCZNP762GXLNW36DRQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340038290 (553h16m45.211121335s) [noise=true , address=145.79.227.14:4839 , piece_id=BSK4K6LL7EW2QATY54QTMUEIWUQ6VPWFMPEW3UPQS72QPLL6CMYQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340294728 (553h16m45.211377773s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340324325 (553h16m45.21140737s) [noise=true , address=145.79.227.42:4121 , piece_id=DAQ6KEYTIWDER4GAV6GFVHFZ5KEVUXTGSM2S2Q25GRZTJFTC3NWA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.341623909 (553h16m45.212706944s) [noise=true , address=45.91.116.8:4046 , piece_id=XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:47.048374882 (553h16m46.919457917s) [upload=485084]: Next returned nil segment Dec 28 16:45:47.048386494 (553h16m46.919469529s) [upload=485084]: Waiting for error group managing segments... Dec 28 16:45:47.334974136 (553h16m47.206057171s) [noise=true , address=145.79.227.19:4078 , piece_id=VQMJIONT6M3YTEXBQ6KNILMFG7SS7WNZ2TTHZSMS7TGU47U33C2A, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.335852137 (553h16m47.206935182s) [noise=true , address=145.79.227.32:4836 , piece_id=2MDYMLYUAVLK3HK4KWWJWFOE6LG5KLAIT7QCZNP762GXLNW36DRQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.344227265 (553h16m47.2153103s) [noise=true , address=145.79.227.10:4401 , piece_id=R5R4MZQSPATWYWNLRGGUCMCUSEMHLVQQGOJWBPI3DQYSSUZPANSQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.346648114 (553h16m47.21773115s) [noise=true , address=145.79.227.7:4295 , piece_id=TV4LICWPQ4RL4XFZ55ZET6OLVADDZDKFS22NTPBCSHGPKV6S6DIQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.347586722 (553h16m47.218669767s) [noise=true , address=145.79.227.3:4151 , piece_id=TQI7V76ULX6YLDBDWNNCTXHTWTF3P2SYGLHPN5635M52VLZUJRSQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.348580264 (553h16m47.219663309s) [noise=true , address=145.79.227.27:4690 , piece_id=6ZHH6H7ULO3A5WK7USEYM3RJFSKZDBXHI7YUZYFMXVRM2P5YJVIQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.364151624 (553h16m47.235234709s) [noise=true , address=145.79.227.51:4410 , piece_id=JZ4V6ID5WY35OK25C5SWTC7LCMY3AZ6J4LNLADFFWUKUITM67FDQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.365004788 (553h16m47.236087823s) [noise=true , address=145.79.227.21:4473 , piece_id=7EXSPSXDE5Q2DD2KMZFB765BYKUBUNI4A6C5KDJLQ7SEXPK4ONCQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.366190105 (553h16m47.23727314s) [noise=true , address=145.79.227.38:4979 , piece_id=MNO6BEYQ7Y2Y4CYQ63QV4RRU5W7SMDK2LGG6JRUZ6IV45NK3QMOQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.368810305 (553h16m47.239893361s) [noise=true , address=145.79.227.42:4121 , piece_id=DAQ6KEYTIWDER4GAV6GFVHFZ5KEVUXTGSM2S2Q25GRZTJFTC3NWA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.512574536 (553h16m47.383657582s) [noise=true , address=77.90.31.10:4519 , piece_id=7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.536220132 (553h16m47.407303167s) [noise=true , address=145.79.227.14:4839 , piece_id=BSK4K6LL7EW2QATY54QTMUEIWUQ6VPWFMPEW3UPQS72QPLL6CMYQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.564997609 (553h16m47.436080654s) [noise=true , address=45.91.116.8:4046 , piece_id=XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.565016785 (553h16m47.43609982s) [seg_pos={0 0}, upload=485084]: Segment reached optimal threshold of 13 pieces. Dec 28 16:45:47.565204964 (553h16m47.436287999s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection Dec 28 16:45:47.565211506 (553h16m47.436294541s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565257914 (553h16m47.436340949s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 16:45:47.565265890 (553h16m47.436348925s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565418040 (553h16m47.436501085s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 16:45:47.565424452 (553h16m47.436507487s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565438438 (553h16m47.436521473s) [seg_pos={0 0}, upload=485084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:45:47.565506668 (553h16m47.436589703s) [upload=485084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:45:47.565511137 (553h16m47.436594182s) [upload=485084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:45:47.854627794 (553h16m47.725710839s) [upload=485084]: Done uploading segments... Dec 28 16:45:47.854653423 (553h16m47.725736468s) [upload=485084]: Upload finished. err: <nil> Duration: 5.995449119s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X.] duration: 3.628209715s trace: https://jaeger.dev.storj.io/trace/1ab9abc21447e609 FAIL (2025-12-28 16:46:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.628237808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XXX] duration: 30.001764082s trace: https://jaeger.dev.storj.io/trace/3d51b27ec0a0f509 FAIL (2025-12-28 16:46:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA) context deadline exceeded ======LOGS====== Dec 28 16:45:45.080572784 (553h16m44.951655829s) [upload=485084]: Starting upload Dec 28 16:45:45.080625815 (553h16m44.95170885s) [upload=485084]: Done starting upload Dec 28 16:45:45.080832970 (553h16m44.951916005s) [upload=485084]: Uploading segments... Dec 28 16:45:45.080876843 (553h16m44.951959878s) [upload=485084]: Got next segment. Inline: false Dec 28 16:45:45.080914545 (553h16m44.95199758s) [upload=485084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:45:45.080918873 (553h16m44.952001908s) [upload=485084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:45:45.337769280 (553h16m45.208852315s) [seg_pos={0 0}, upload=485084]: Begin upload segment... Dec 28 16:45:45.337924525 (553h16m45.20900757s) [seg_pos={0 0}, upload=485084]: Done begin upload segment. Dec 28 16:45:45.338005900 (553h16m45.209088935s) [noise=true , address=145.79.227.3:4151 , piece_id=TQI7V76ULX6YLDBDWNNCTXHTWTF3P2SYGLHPN5635M52VLZUJRSQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338024616 (553h16m45.209107661s) [noise=true , address=145.79.227.19:4078 , piece_id=VQMJIONT6M3YTEXBQ6KNILMFG7SS7WNZ2TTHZSMS7TGU47U33C2A, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338393178 (553h16m45.209476213s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338653052 (553h16m45.209736087s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.338915873 (553h16m45.209998908s) [noise=true , address=145.79.227.51:4410 , piece_id=JZ4V6ID5WY35OK25C5SWTC7LCMY3AZ6J4LNLADFFWUKUITM67FDQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339188031 (553h16m45.210271066s) [noise=true , address=145.79.227.7:4295 , piece_id=TV4LICWPQ4RL4XFZ55ZET6OLVADDZDKFS22NTPBCSHGPKV6S6DIQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339195806 (553h16m45.210278841s) [noise=true , address=77.90.31.10:4519 , piece_id=7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339459137 (553h16m45.210542182s) [noise=true , address=145.79.227.38:4979 , piece_id=MNO6BEYQ7Y2Y4CYQ63QV4RRU5W7SMDK2LGG6JRUZ6IV45NK3QMOQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339494494 (553h16m45.21057753s) [noise=true , address=145.79.227.27:4690 , piece_id=6ZHH6H7ULO3A5WK7USEYM3RJFSKZDBXHI7YUZYFMXVRM2P5YJVIQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339745392 (553h16m45.210828437s) [noise=true , address=145.79.227.10:4401 , piece_id=R5R4MZQSPATWYWNLRGGUCMCUSEMHLVQQGOJWBPI3DQYSSUZPANSQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.339769458 (553h16m45.210852503s) [noise=true , address=145.79.227.21:4473 , piece_id=7EXSPSXDE5Q2DD2KMZFB765BYKUBUNI4A6C5KDJLQ7SEXPK4ONCQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340019815 (553h16m45.21110285s) [noise=true , address=145.79.227.32:4836 , piece_id=2MDYMLYUAVLK3HK4KWWJWFOE6LG5KLAIT7QCZNP762GXLNW36DRQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340038290 (553h16m45.211121335s) [noise=true , address=145.79.227.14:4839 , piece_id=BSK4K6LL7EW2QATY54QTMUEIWUQ6VPWFMPEW3UPQS72QPLL6CMYQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340294728 (553h16m45.211377773s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.340324325 (553h16m45.21140737s) [noise=true , address=145.79.227.42:4121 , piece_id=DAQ6KEYTIWDER4GAV6GFVHFZ5KEVUXTGSM2S2Q25GRZTJFTC3NWA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:45.341623909 (553h16m45.212706944s) [noise=true , address=45.91.116.8:4046 , piece_id=XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA, seg_pos={0 0}, upload=485084]: Uploading piece... Dec 28 16:45:47.048374882 (553h16m46.919457917s) [upload=485084]: Next returned nil segment Dec 28 16:45:47.048386494 (553h16m46.919469529s) [upload=485084]: Waiting for error group managing segments... Dec 28 16:45:47.334974136 (553h16m47.206057171s) [noise=true , address=145.79.227.19:4078 , piece_id=VQMJIONT6M3YTEXBQ6KNILMFG7SS7WNZ2TTHZSMS7TGU47U33C2A, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.335852137 (553h16m47.206935182s) [noise=true , address=145.79.227.32:4836 , piece_id=2MDYMLYUAVLK3HK4KWWJWFOE6LG5KLAIT7QCZNP762GXLNW36DRQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.344227265 (553h16m47.2153103s) [noise=true , address=145.79.227.10:4401 , piece_id=R5R4MZQSPATWYWNLRGGUCMCUSEMHLVQQGOJWBPI3DQYSSUZPANSQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.346648114 (553h16m47.21773115s) [noise=true , address=145.79.227.7:4295 , piece_id=TV4LICWPQ4RL4XFZ55ZET6OLVADDZDKFS22NTPBCSHGPKV6S6DIQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.347586722 (553h16m47.218669767s) [noise=true , address=145.79.227.3:4151 , piece_id=TQI7V76ULX6YLDBDWNNCTXHTWTF3P2SYGLHPN5635M52VLZUJRSQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.348580264 (553h16m47.219663309s) [noise=true , address=145.79.227.27:4690 , piece_id=6ZHH6H7ULO3A5WK7USEYM3RJFSKZDBXHI7YUZYFMXVRM2P5YJVIQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.364151624 (553h16m47.235234709s) [noise=true , address=145.79.227.51:4410 , piece_id=JZ4V6ID5WY35OK25C5SWTC7LCMY3AZ6J4LNLADFFWUKUITM67FDQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.365004788 (553h16m47.236087823s) [noise=true , address=145.79.227.21:4473 , piece_id=7EXSPSXDE5Q2DD2KMZFB765BYKUBUNI4A6C5KDJLQ7SEXPK4ONCQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.366190105 (553h16m47.23727314s) [noise=true , address=145.79.227.38:4979 , piece_id=MNO6BEYQ7Y2Y4CYQ63QV4RRU5W7SMDK2LGG6JRUZ6IV45NK3QMOQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.368810305 (553h16m47.239893361s) [noise=true , address=145.79.227.42:4121 , piece_id=DAQ6KEYTIWDER4GAV6GFVHFZ5KEVUXTGSM2S2Q25GRZTJFTC3NWA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.512574536 (553h16m47.383657582s) [noise=true , address=77.90.31.10:4519 , piece_id=7R4CMLJVWENHHXHGTCIM5ZTK4666MOEW26C3KJN6CCPUTAYNB3GA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.536220132 (553h16m47.407303167s) [noise=true , address=145.79.227.14:4839 , piece_id=BSK4K6LL7EW2QATY54QTMUEIWUQ6VPWFMPEW3UPQS72QPLL6CMYQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.564997609 (553h16m47.436080654s) [noise=true , address=45.91.116.8:4046 , piece_id=XAH53C52J2XHPSKKHTRND4L4KVWMO5OHUQVUDD5L5K5OCVMANNHA, seg_pos={0 0}, upload=485084]: Done uploading piece. err: <nil> Dec 28 16:45:47.565016785 (553h16m47.43609982s) [seg_pos={0 0}, upload=485084]: Segment reached optimal threshold of 13 pieces. Dec 28 16:45:47.565204964 (553h16m47.436287999s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57498->45.91.116.11:4118: use of closed network connection Dec 28 16:45:47.565211506 (553h16m47.436294541s) [noise=true , address=45.91.116.11:4118 , piece_id=3O45JVD6HVC2EKTVDRMZG6XJFCWTOPEHY65N4XWJR4FDNW2KEH7Q, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565257914 (553h16m47.436340949s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 16:45:47.565265890 (553h16m47.436348925s) [noise=true , address=77.90.31.16:4735 , piece_id=IL2QFUMYVSHCJGUCX2CJ6UFT2RRYZPN3LQAEFA2YKZSERETBG52Q, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565418040 (553h16m47.436501085s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 16:45:47.565424452 (553h16m47.436507487s) [noise=true , address=45.91.116.12:4227 , piece_id=72KNPHQ4LHSLVJTDYTD5CTHYZCIL7CXG76OBDRQBQWODIOCYO5GQ, seg_pos={0 0}, upload=485084]: Long tail cancellation due to optimal threshold Dec 28 16:45:47.565438438 (553h16m47.436521473s) [seg_pos={0 0}, upload=485084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:45:47.565506668 (553h16m47.436589703s) [upload=485084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:45:47.565511137 (553h16m47.436594182s) [upload=485084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:45:47.854627794 (553h16m47.725710839s) [upload=485084]: Done uploading segments... Dec 28 16:45:47.854653423 (553h16m47.725736468s) [upload=485084]: Upload finished. err: <nil> Duration: 30.001787406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.............X.........................X............] duration: 46.450021974s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:45:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 46.450049097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....X] duration: 5.002364698s trace: https://jaeger.dev.storj.io/trace/1ab9abc21447e609 FAIL (2025-12-28 16:45:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002388563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.XX] duration: 30.002083748s trace: https://jaeger.dev.storj.io/trace/466c3523ae68f0f4 FAIL (2025-12-28 16:45:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 16:44:50.273436342 (553h15m50.144519377s) [upload=485072]: Starting upload Dec 28 16:44:50.273486949 (553h15m50.144569984s) [upload=485072]: Done starting upload Dec 28 16:44:50.273576018 (553h15m50.144659053s) [upload=485072]: Uploading segments... Dec 28 16:44:50.273655169 (553h15m50.144738204s) [upload=485072]: Got next segment. Inline: false Dec 28 16:44:50.273683222 (553h15m50.144766257s) [upload=485072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:44:50.273689243 (553h15m50.144772278s) [upload=485072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:44:50.514880431 (553h15m50.385963476s) [seg_pos={0 0}, upload=485072]: Begin upload segment... Dec 28 16:44:50.514924966 (553h15m50.386008001s) [seg_pos={0 0}, upload=485072]: Done begin upload segment. Dec 28 16:44:50.514948230 (553h15m50.386031265s) [noise=true , address=145.79.227.11:4731 , piece_id=FFZC7SDRUPKZ3E274EUQO4UULDQYC6K5GV2NCW5RNS3XZCMENQ7Q, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.514987365 (553h15m50.3860704s) [noise=true , address=145.79.227.45:4230 , piece_id=ZYGGPHJWKPQMDQEN2DYUI2RLJRO3W7D2QJNF2PSWHAIHJVP2QUAA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.515392587 (553h15m50.386475622s) [noise=true , address=145.79.227.38:4978 , piece_id=CIKR4VW2MLBAKC5LZUWKMDOLMOCYPUMWKNJDY55DY4MHJBCVCCEA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.515666809 (553h15m50.386749844s) [noise=true , address=45.91.116.13:4154 , piece_id=YXTVBIO57JD4UAF76NFFDAISJIK7LRXJNQA4BEYEMF6F3L24X3SQ, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.515940110 (553h15m50.387023145s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHDOSBGDS4BCZHG7GEQZBSKQ3ITMAOUOVGC5HHKYUTRLSM2F22EQ, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.516238207 (553h15m50.387321242s) [noise=true , address=145.79.227.52:4447 , piece_id=6VNGFKEXO2XY6T6VY6RNFL4KA32BXENELBYRNKOLNBRQ47A4YCJA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.516502130 (553h15m50.387585165s) [noise=true , address=45.91.116.4:4940 , piece_id=EH77CDJKYUXIILYSUKNKNO4SYI7UTE62HZZERI4TOMLQVBEHGE4A, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.516776562 (553h15m50.387859597s) [noise=true , address=45.91.116.7:4010 , piece_id=ZFLDTTCVQ77Q2UKPLJCU3W6KFMP6A3QZRYX774WVA3KGXED57FCA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.517077545 (553h15m50.38816058s) [noise=true , address=145.79.227.4:4186 , piece_id=ZHDWG24ZQGZTB6KTCCINCT2TVAGZBSIIYFAMZSHAVOYM3DQ5JNQA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.517385151 (553h15m50.388468196s) [noise=true , address=145.79.227.43:4157 , piece_id=SLDMC5EIKKBP26QBCUVYO6RNRW4DSGBW7R4K7VZC2JSVOJRDXDMA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.518502278 (553h15m50.389585313s) [noise=true , address=145.79.227.2:4113 , piece_id=6C2SS5GEMNM7ZZEWB2252H4GHIGUV4JROBQ7ROAYGXZ7SFJPJLZA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.519630076 (553h15m50.390713121s) [noise=true , address=77.90.31.18:4808 , piece_id=ZUDGWTV7JPW4QDR3JGEFDCVEOL7ANPJVQ6GSRRVXD2LDX6L63T7A, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.520741121 (553h15m50.391824156s) [noise=true , address=145.79.227.10:4401 , piece_id=5253ZF4K7DOYO2ERNM4EP5SQDUR3QZ3PJTJAOWBMGO3DRHXLKKWQ, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.521827440 (553h15m50.392910475s) [noise=true , address=145.79.227.5:4223 , piece_id=UQI22477FSEDIPK7DCASOUY2LZUUHT5ZB2NMEHWTJMUAOD47LKIQ, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.523115723 (553h15m50.394198758s) [noise=true , address=77.90.31.10:4519 , piece_id=MBLID7QTOW5DWD3AOQTSLL7KZE3LNI4MVNZ3RBWOATWJYNEATFXA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:50.525599562 (553h15m50.396682607s) [noise=true , address=45.91.116.6:4975 , piece_id=NALVPPL7NM5ZFXQHNALZLEVDVIW64H7EKAMWRUUVLWCHNSUSNSUA, seg_pos={0 0}, upload=485072]: Uploading piece... Dec 28 16:44:52.140391579 (553h15m52.011474624s) [upload=485072]: Next returned nil segment Dec 28 16:44:52.140408592 (553h15m52.011491627s) [upload=485072]: Waiting for error group managing segments... Dec 28 16:44:52.522806051 (553h15m52.393889086s) [noise=true , address=145.79.227.11:4731 , piece_id=FFZC7SDRUPKZ3E274EUQO4UULDQYC6K5GV2NCW5RNS3XZCMENQ7Q, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.523928959 (553h15m52.395012004s) [noise=true , address=145.79.227.52:4447 , piece_id=6VNGFKEXO2XY6T6VY6RNFL4KA32BXENELBYRNKOLNBRQ47A4YCJA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.527114495 (553h15m52.39819754s) [noise=true , address=145.79.227.2:4113 , piece_id=6C2SS5GEMNM7ZZEWB2252H4GHIGUV4JROBQ7ROAYGXZ7SFJPJLZA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.537217453 (553h15m52.408300498s) [noise=true , address=145.79.227.45:4230 , piece_id=ZYGGPHJWKPQMDQEN2DYUI2RLJRO3W7D2QJNF2PSWHAIHJVP2QUAA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.543897222 (553h15m52.414980267s) [noise=true , address=145.79.227.38:4978 , piece_id=CIKR4VW2MLBAKC5LZUWKMDOLMOCYPUMWKNJDY55DY4MHJBCVCCEA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.548308112 (553h15m52.419391157s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHDOSBGDS4BCZHG7GEQZBSKQ3ITMAOUOVGC5HHKYUTRLSM2F22EQ, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.549245086 (553h15m52.420328131s) [noise=true , address=145.79.227.10:4401 , piece_id=5253ZF4K7DOYO2ERNM4EP5SQDUR3QZ3PJTJAOWBMGO3DRHXLKKWQ, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.550000414 (553h15m52.421083469s) [noise=true , address=145.79.227.5:4223 , piece_id=UQI22477FSEDIPK7DCASOUY2LZUUHT5ZB2NMEHWTJMUAOD47LKIQ, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.550793594 (553h15m52.421876639s) [noise=true , address=145.79.227.4:4186 , piece_id=ZHDWG24ZQGZTB6KTCCINCT2TVAGZBSIIYFAMZSHAVOYM3DQ5JNQA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.764499557 (553h15m52.635582592s) [noise=true , address=45.91.116.13:4154 , piece_id=YXTVBIO57JD4UAF76NFFDAISJIK7LRXJNQA4BEYEMF6F3L24X3SQ, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.779158740 (553h15m52.650241775s) [noise=true , address=45.91.116.6:4975 , piece_id=NALVPPL7NM5ZFXQHNALZLEVDVIW64H7EKAMWRUUVLWCHNSUSNSUA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.813043823 (553h15m52.684126868s) [noise=true , address=77.90.31.10:4519 , piece_id=MBLID7QTOW5DWD3AOQTSLL7KZE3LNI4MVNZ3RBWOATWJYNEATFXA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.927094946 (553h15m52.798177981s) [noise=true , address=77.90.31.18:4808 , piece_id=ZUDGWTV7JPW4QDR3JGEFDCVEOL7ANPJVQ6GSRRVXD2LDX6L63T7A, seg_pos={0 0}, upload=485072]: Done uploading piece. err: <nil> Dec 28 16:44:52.927120685 (553h15m52.79820372s) [seg_pos={0 0}, upload=485072]: Segment reached optimal threshold of 13 pieces. Dec 28 16:44:52.927401710 (553h15m52.798484745s) [noise=true , address=45.91.116.4:4940 , piece_id=EH77CDJKYUXIILYSUKNKNO4SYI7UTE62HZZERI4TOMLQVBEHGE4A, seg_pos={0 0}, upload=485072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 16:44:52.927409375 (553h15m52.79849241s) [noise=true , address=45.91.116.4:4940 , piece_id=EH77CDJKYUXIILYSUKNKNO4SYI7UTE62HZZERI4TOMLQVBEHGE4A, seg_pos={0 0}, upload=485072]: Long tail cancellation due to optimal threshold Dec 28 16:44:52.929141944 (553h15m52.800224999s) [noise=true , address=45.91.116.7:4010 , piece_id=ZFLDTTCVQ77Q2UKPLJCU3W6KFMP6A3QZRYX774WVA3KGXED57FCA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 28 16:44:52.929151712 (553h15m52.800234747s) [noise=true , address=45.91.116.7:4010 , piece_id=ZFLDTTCVQ77Q2UKPLJCU3W6KFMP6A3QZRYX774WVA3KGXED57FCA, seg_pos={0 0}, upload=485072]: Long tail cancellation due to optimal threshold Dec 28 16:44:52.929316326 (553h15m52.800399361s) [noise=true , address=145.79.227.43:4157 , piece_id=SLDMC5EIKKBP26QBCUVYO6RNRW4DSGBW7R4K7VZC2JSVOJRDXDMA, seg_pos={0 0}, upload=485072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 28 16:44:52.929322858 (553h15m52.800405903s) [noise=true , address=145.79.227.43:4157 , piece_id=SLDMC5EIKKBP26QBCUVYO6RNRW4DSGBW7R4K7VZC2JSVOJRDXDMA, seg_pos={0 0}, upload=485072]: Long tail cancellation due to optimal threshold Dec 28 16:44:52.929340582 (553h15m52.800423617s) [seg_pos={0 0}, upload=485072]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:44:52.929404974 (553h15m52.800488009s) [upload=485072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:44:52.929410915 (553h15m52.800493951s) [upload=485072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:44:53.220603916 (553h15m53.091686951s) [upload=485072]: Done uploading segments... Dec 28 16:44:53.220624064 (553h15m53.091707099s) [upload=485072]: Upload finished. err: <nil> Duration: 30.002104617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X....] duration: 3.490467729s trace: https://jaeger.dev.storj.io/trace/c534c57f90f8f28 FAIL (2025-12-28 16:44:32 UTC) orbiter warning: context deadline exceeded Duration: 3.490498388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.X] duration: 30.002080824s trace: https://jaeger.dev.storj.io/trace/3870ce1bb2cf2ac5 FAIL (2025-12-28 16:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 16:44:02.687975724 (553h15m2.559058769s) [upload=485059]: Starting upload Dec 28 16:44:02.688058332 (553h15m2.559141367s) [upload=485059]: Done starting upload Dec 28 16:44:02.688106153 (553h15m2.559189188s) [upload=485059]: Uploading segments... Dec 28 16:44:02.688316343 (553h15m2.559399388s) [upload=485059]: Got next segment. Inline: false Dec 28 16:44:02.688371057 (553h15m2.559454103s) [upload=485059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:44:02.688377419 (553h15m2.559460465s) [upload=485059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:44:02.938871232 (553h15m2.809954287s) [seg_pos={0 0}, upload=485059]: Begin upload segment... Dec 28 16:44:02.938937318 (553h15m2.810020363s) [seg_pos={0 0}, upload=485059]: Done begin upload segment. Dec 28 16:44:02.938983696 (553h15m2.810066751s) [noise=true , address=145.79.227.17:4972 , piece_id=OWBN2GFYCX6EA6XM6QD4D6QBXSYEZMWO4KMPDWC2777PNPCKJRSA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.939006900 (553h15m2.810089946s) [noise=true , address=45.91.116.8:4046 , piece_id=PHLLSA5TDII3FOX6APAOQOZVZC72G3DZD2G5ZJMLH4YGMTV2ZNKQ, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.939070671 (553h15m2.810153706s) [noise=true , address=77.90.31.14:4661 , piece_id=6HMCEW5DBK7KYNCIKOW34WBH6RFDPFUBGULP27TGTTPEIH7LHPYA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.939902906 (553h15m2.810985961s) [noise=true , address=45.91.116.12:4228 , piece_id=HEJQ5FT2LC3YSO5RUBPB2CSN3GN7JWX4Z77TNIXAJ3EFTSAYY77A, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.940367732 (553h15m2.811450777s) [noise=true , address=145.79.227.4:4187 , piece_id=LU6ZBKPZCZVR7SSA6GK7WED57LYUBUW4QKJ65PBOTMHURWGITMRA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.940799253 (553h15m2.811882298s) [noise=true , address=77.90.31.11:4556 , piece_id=NKQY3I5UNPARH2TCO75JHUWTQUTTU6TI3ENPJOS6KD6IWEELZJIA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.941225825 (553h15m2.81230887s) [noise=true , address=145.79.227.37:4566 , piece_id=FMLRDHPRYZBU45CHBHR3KASI2QVPVCAIIE7S5GBY7J4R5U4TWLCQ, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.941776113 (553h15m2.812859158s) [noise=true , address=145.79.227.14:4839 , piece_id=KOGOKNUTAE3TFA2WK6ZBHTZN5AODK2ZGNOF77WGRVQLY7HOFDCSQ, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.942442662 (553h15m2.813525707s) [noise=true , address=145.79.227.50:4373 , piece_id=K6X5PQHKFUCZUHRQS5536LNC3RR4CHYNWRZIH6GIQFBRZ4OAXNRA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.942933897 (553h15m2.814016942s) [noise=true , address=145.79.227.30:4762 , piece_id=T447BABYTPCAFRWDV2VZC6H23S55TUD3HFQT24ZMNSZG4MKFV5ZA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.943595708 (553h15m2.814678753s) [noise=true , address=45.91.116.5:4298 , piece_id=QXKXY5ZZBMIWZY4BKDVXUYAEURJ45WMW2CNW5FAISY46QMWTDD5Q, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.945271630 (553h15m2.816354675s) [noise=true , address=145.79.227.27:4690 , piece_id=H5ZQVF3S3DRUGDZ75KIP4KIGUR4EPGV665MTBJ6MTIUA6CRGHMCA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.945612027 (553h15m2.816695082s) [noise=true , address=145.79.227.5:4221 , piece_id=NXNL5LDU3WZQVTW74TW55RAFNQWU7JW2G6P7LK5NXWIBRXK2IILQ, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.947488120 (553h15m2.818571165s) [noise=true , address=145.79.227.19:4078 , piece_id=6OJMAOSVGP2DVW3KJLL7NRP4PYTYAHLATOUMH553RDYKLOJREWDQ, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.947651281 (553h15m2.818734366s) [noise=true , address=145.79.227.52:4446 , piece_id=NNN4HNZOI72R2OKHOF2A5QZAFUEQINZXS5VFCKKFYROSVXQ4P7FA, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:02.949503617 (553h15m2.820586673s) [noise=true , address=45.91.116.11:4117 , piece_id=7MNOUA3Y7AZGRKJVKG74QG5ED3EZN7LBOQQA66WRYA3Z5NARVNVQ, seg_pos={0 0}, upload=485059]: Uploading piece... Dec 28 16:44:04.662731183 (553h15m4.533814228s) [upload=485059]: Next returned nil segment Dec 28 16:44:04.662743145 (553h15m4.533826191s) [upload=485059]: Waiting for error group managing segments... Dec 28 16:44:04.938693862 (553h15m4.809776907s) [noise=true , address=145.79.227.52:4446 , piece_id=NNN4HNZOI72R2OKHOF2A5QZAFUEQINZXS5VFCKKFYROSVXQ4P7FA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:04.949769422 (553h15m4.820852467s) [noise=true , address=145.79.227.17:4972 , piece_id=OWBN2GFYCX6EA6XM6QD4D6QBXSYEZMWO4KMPDWC2777PNPCKJRSA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:04.956086251 (553h15m4.827169286s) [noise=true , address=145.79.227.5:4221 , piece_id=NXNL5LDU3WZQVTW74TW55RAFNQWU7JW2G6P7LK5NXWIBRXK2IILQ, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:04.965455070 (553h15m4.836538115s) [noise=true , address=145.79.227.50:4373 , piece_id=K6X5PQHKFUCZUHRQS5536LNC3RR4CHYNWRZIH6GIQFBRZ4OAXNRA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:04.979733990 (553h15m4.850817035s) [noise=true , address=145.79.227.37:4566 , piece_id=FMLRDHPRYZBU45CHBHR3KASI2QVPVCAIIE7S5GBY7J4R5U4TWLCQ, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:04.981122854 (553h15m4.852205899s) [noise=true , address=145.79.227.27:4690 , piece_id=H5ZQVF3S3DRUGDZ75KIP4KIGUR4EPGV665MTBJ6MTIUA6CRGHMCA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:04.981257070 (553h15m4.852340105s) [noise=true , address=145.79.227.19:4078 , piece_id=6OJMAOSVGP2DVW3KJLL7NRP4PYTYAHLATOUMH553RDYKLOJREWDQ, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:04.991344839 (553h15m4.862427884s) [noise=true , address=145.79.227.14:4839 , piece_id=KOGOKNUTAE3TFA2WK6ZBHTZN5AODK2ZGNOF77WGRVQLY7HOFDCSQ, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:05.000304960 (553h15m4.871388015s) [noise=true , address=145.79.227.4:4187 , piece_id=LU6ZBKPZCZVR7SSA6GK7WED57LYUBUW4QKJ65PBOTMHURWGITMRA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:05.122786446 (553h15m4.993869491s) [noise=true , address=77.90.31.11:4556 , piece_id=NKQY3I5UNPARH2TCO75JHUWTQUTTU6TI3ENPJOS6KD6IWEELZJIA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:05.286129178 (553h15m5.157212223s) [noise=true , address=45.91.116.8:4046 , piece_id=PHLLSA5TDII3FOX6APAOQOZVZC72G3DZD2G5ZJMLH4YGMTV2ZNKQ, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:05.331032741 (553h15m5.202115797s) [noise=true , address=45.91.116.5:4298 , piece_id=QXKXY5ZZBMIWZY4BKDVXUYAEURJ45WMW2CNW5FAISY46QMWTDD5Q, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:05.460915861 (553h15m5.331998906s) [noise=true , address=45.91.116.12:4228 , piece_id=HEJQ5FT2LC3YSO5RUBPB2CSN3GN7JWX4Z77TNIXAJ3EFTSAYY77A, seg_pos={0 0}, upload=485059]: Done uploading piece. err: <nil> Dec 28 16:44:05.460941641 (553h15m5.332024686s) [seg_pos={0 0}, upload=485059]: Segment reached optimal threshold of 13 pieces. Dec 28 16:44:05.461189472 (553h15m5.332272517s) [noise=true , address=77.90.31.14:4661 , piece_id=6HMCEW5DBK7KYNCIKOW34WBH6RFDPFUBGULP27TGTTPEIH7LHPYA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 16:44:05.461198721 (553h15m5.332281766s) [noise=true , address=77.90.31.14:4661 , piece_id=6HMCEW5DBK7KYNCIKOW34WBH6RFDPFUBGULP27TGTTPEIH7LHPYA, seg_pos={0 0}, upload=485059]: Long tail cancellation due to optimal threshold Dec 28 16:44:05.461279144 (553h15m5.332362189s) [noise=true , address=45.91.116.11:4117 , piece_id=7MNOUA3Y7AZGRKJVKG74QG5ED3EZN7LBOQQA66WRYA3Z5NARVNVQ, seg_pos={0 0}, upload=485059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:49270->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49270->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49270->45.91.116.11:4117: use of closed network connection Dec 28 16:44:05.461287870 (553h15m5.332370915s) [noise=true , address=45.91.116.11:4117 , piece_id=7MNOUA3Y7AZGRKJVKG74QG5ED3EZN7LBOQQA66WRYA3Z5NARVNVQ, seg_pos={0 0}, upload=485059]: Long tail cancellation due to optimal threshold Dec 28 16:44:05.461585387 (553h15m5.332668462s) [noise=true , address=145.79.227.30:4762 , piece_id=T447BABYTPCAFRWDV2VZC6H23S55TUD3HFQT24ZMNSZG4MKFV5ZA, seg_pos={0 0}, upload=485059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 16:44:05.461595295 (553h15m5.332678331s) [noise=true , address=145.79.227.30:4762 , piece_id=T447BABYTPCAFRWDV2VZC6H23S55TUD3HFQT24ZMNSZG4MKFV5ZA, seg_pos={0 0}, upload=485059]: Long tail cancellation due to optimal threshold Dec 28 16:44:05.461619923 (553h15m5.332702968s) [seg_pos={0 0}, upload=485059]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:44:05.461710184 (553h15m5.332793219s) [upload=485059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:44:05.461716426 (553h15m5.332799471s) [upload=485059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:44:05.758622956 (553h15m5.629705991s) [upload=485059]: Done uploading segments... Dec 28 16:44:05.758643565 (553h15m5.6297266s) [upload=485059]: Upload finished. err: <nil> Duration: 30.002104117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.............X.........................X...........] duration: 20.552715474s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:44:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 20.552735832s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [.X.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X...] duration: 3.38800287s trace: https://jaeger.dev.storj.io/trace/c534c57f90f8f28 FAIL (2025-12-28 16:43:58 UTC) orbiter warning: context deadline exceeded Duration: 3.388025222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.............X.........................X..........] duration: 39.340317967s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:43:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 39.340346522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X.] duration: 5.929029325s trace: https://jaeger.dev.storj.io/trace/519a26b1f03b2094 FAIL (2025-12-28 16:43:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 16:42:45.032946719 (553h13m44.904029764s) [upload=485040]: Starting upload Dec 28 16:42:45.033058151 (553h13m44.904141196s) [upload=485040]: Done starting upload Dec 28 16:42:45.033190033 (553h13m44.904273078s) [upload=485040]: Uploading segments... Dec 28 16:42:45.033448554 (553h13m44.9045316s) [upload=485040]: Got next segment. Inline: false Dec 28 16:42:45.033482599 (553h13m44.904565644s) [upload=485040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:42:45.033488040 (553h13m44.904571075s) [upload=485040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:42:45.276784437 (553h13m45.147867482s) [seg_pos={0 0}, upload=485040]: Begin upload segment... Dec 28 16:42:45.276839502 (553h13m45.147922537s) [seg_pos={0 0}, upload=485040]: Done begin upload segment. Dec 28 16:42:45.276877514 (553h13m45.147960559s) [noise=true , address=145.79.227.14:4840 , piece_id=5ZNNFOAOCO3VGTOTA6AB4MUMMDGS37XIBEC4C3KVCTVI2FJGZ3XQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.276926738 (553h13m45.148009773s) [noise=true , address=145.79.227.38:4977 , piece_id=J4SWU3AIQBLTZQH7232LNKGNONJSAKKKLWR7KRSZDF7SCJVLZL6A, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.277413815 (553h13m45.14849686s) [noise=true , address=45.91.116.11:4119 , piece_id=WJFBON3R2X5IZABBTUYEEHQPUKR5P3QDAXT26MF6Q76X36TFISNA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278080174 (553h13m45.149163209s) [noise=true , address=77.90.31.17:4773 , piece_id=F47V7WJ5EECOSNENXTFDLNYX7YWEP732SGXCNE5NHHGFLE526APQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278441913 (553h13m45.149524948s) [noise=true , address=145.79.227.37:4567 , piece_id=ZSZBID2D234S4WRH53PHB5RBMROBWRBLG3VTEQ7RVFD4SUM4X4KQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278577892 (553h13m45.149660937s) [noise=true , address=45.91.116.7:4012 , piece_id=QGW5EMYKZJ2GCG6ZG53TXFUGXYOFLQFCGXU5QTUU7CKMVASEAEUA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278750060 (553h13m45.149833095s) [noise=true , address=45.91.116.5:4299 , piece_id=DBJ4S3XAAZT7TUI3DBH53GXOIOLXM3MM37BOCKHGOKCVBWMYIVOA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279069097 (553h13m45.150152143s) [noise=true , address=145.79.227.11:4731 , piece_id=EQ5BDZZL65OO7VJQWC357TK6DK53423FRLEYO3TDS3O7XIQIGPHA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279402301 (553h13m45.150485346s) [noise=true , address=77.90.31.18:4805 , piece_id=CY5HMDT2ECKE5O7JKFETTNWOADX3GTKBXJPDSHXX3R2U3AKDF57A, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279672576 (553h13m45.150755621s) [noise=true , address=45.91.116.14:4190 , piece_id=VWPU4XX4VRORTASUQ6EJ2YCW2L5SKMFAX4FGZMUBVX2FYOBXCORQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279960294 (553h13m45.151043339s) [noise=true , address=145.79.227.47:4266 , piece_id=M4FZPADV4BFVZ5Z7NPN7QGJUNYEB2DJMSDALH6DAS3PXTNHFVJUQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.280260987 (553h13m45.151344022s) [noise=true , address=145.79.227.42:4123 , piece_id=BV7DW22FZIW4FBZDUHPYJ4EOARTF2U57HHQ47AUIWQOKQCO2N3XA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.280635951 (553h13m45.151718986s) [noise=true , address=145.79.227.45:4231 , piece_id=6RCTC4GLP42DI4WSC2MFRNMVGXNQ657XOV2DFMGLWHP6U7RD42PQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.280964486 (553h13m45.152047521s) [noise=true , address=45.91.116.13:4156 , piece_id=RETU4J64HWVJKSEE5DJK3LMYKGPEJG4MJR4GDBCR5CQKK7LGMZ2A, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.281145160 (553h13m45.152228205s) [noise=true , address=145.79.227.43:4158 , piece_id=VK3AWLWHVHGLIWFX5BCHIF7QSFMHTBTDBTI2QRUV3ZAKOHJQOTIA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.281829383 (553h13m45.152912428s) [noise=true , address=145.79.227.16:4911 , piece_id=HJFD6SDEX4S7MYFTMXKNS7DR2SBMBVWR432AUYCDUHLSWJAFCAIA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:46.900749110 (553h13m46.771832145s) [upload=485040]: Next returned nil segment Dec 28 16:42:46.900763998 (553h13m46.771847034s) [upload=485040]: Waiting for error group managing segments... Dec 28 16:42:47.280334636 (553h13m47.151417671s) [noise=true , address=145.79.227.45:4231 , piece_id=6RCTC4GLP42DI4WSC2MFRNMVGXNQ657XOV2DFMGLWHP6U7RD42PQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.282940407 (553h13m47.154023452s) [noise=true , address=145.79.227.11:4731 , piece_id=EQ5BDZZL65OO7VJQWC357TK6DK53423FRLEYO3TDS3O7XIQIGPHA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.295013496 (553h13m47.166096531s) [noise=true , address=145.79.227.42:4123 , piece_id=BV7DW22FZIW4FBZDUHPYJ4EOARTF2U57HHQ47AUIWQOKQCO2N3XA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.299210378 (553h13m47.170293413s) [noise=true , address=145.79.227.14:4840 , piece_id=5ZNNFOAOCO3VGTOTA6AB4MUMMDGS37XIBEC4C3KVCTVI2FJGZ3XQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.302133544 (553h13m47.173216589s) [noise=true , address=145.79.227.16:4911 , piece_id=HJFD6SDEX4S7MYFTMXKNS7DR2SBMBVWR432AUYCDUHLSWJAFCAIA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.309764956 (553h13m47.180847991s) [noise=true , address=145.79.227.47:4266 , piece_id=M4FZPADV4BFVZ5Z7NPN7QGJUNYEB2DJMSDALH6DAS3PXTNHFVJUQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.313747769 (553h13m47.184830804s) [noise=true , address=145.79.227.37:4567 , piece_id=ZSZBID2D234S4WRH53PHB5RBMROBWRBLG3VTEQ7RVFD4SUM4X4KQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.324597229 (553h13m47.195680264s) [noise=true , address=145.79.227.38:4977 , piece_id=J4SWU3AIQBLTZQH7232LNKGNONJSAKKKLWR7KRSZDF7SCJVLZL6A, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.470504010 (553h13m47.341587095s) [noise=true , address=77.90.31.17:4773 , piece_id=F47V7WJ5EECOSNENXTFDLNYX7YWEP732SGXCNE5NHHGFLE526APQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.529062365 (553h13m47.4001454s) [noise=true , address=45.91.116.5:4299 , piece_id=DBJ4S3XAAZT7TUI3DBH53GXOIOLXM3MM37BOCKHGOKCVBWMYIVOA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.594172245 (553h13m47.46525528s) [noise=true , address=77.90.31.18:4805 , piece_id=CY5HMDT2ECKE5O7JKFETTNWOADX3GTKBXJPDSHXX3R2U3AKDF57A, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.611947371 (553h13m47.483030406s) [noise=true , address=45.91.116.7:4012 , piece_id=QGW5EMYKZJ2GCG6ZG53TXFUGXYOFLQFCGXU5QTUU7CKMVASEAEUA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.626966491 (553h13m47.498049536s) [noise=true , address=45.91.116.13:4156 , piece_id=RETU4J64HWVJKSEE5DJK3LMYKGPEJG4MJR4GDBCR5CQKK7LGMZ2A, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.626986168 (553h13m47.498069203s) [seg_pos={0 0}, upload=485040]: Segment reached optimal threshold of 13 pieces. Dec 28 16:42:47.627236225 (553h13m47.49831927s) [noise=true , address=45.91.116.14:4190 , piece_id=VWPU4XX4VRORTASUQ6EJ2YCW2L5SKMFAX4FGZMUBVX2FYOBXCORQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:42102->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:42102->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42102->45.91.116.14:4190: use of closed network connection Dec 28 16:42:47.627245873 (553h13m47.498328908s) [noise=true , address=45.91.116.14:4190 , piece_id=VWPU4XX4VRORTASUQ6EJ2YCW2L5SKMFAX4FGZMUBVX2FYOBXCORQ, seg_pos={0 0}, upload=485040]: Long tail cancellation due to optimal threshold Dec 28 16:42:47.627374739 (553h13m47.498457784s) [noise=true , address=145.79.227.43:4158 , piece_id=VK3AWLWHVHGLIWFX5BCHIF7QSFMHTBTDBTI2QRUV3ZAKOHJQOTIA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 16:42:47.627403462 (553h13m47.498486508s) [noise=true , address=145.79.227.43:4158 , piece_id=VK3AWLWHVHGLIWFX5BCHIF7QSFMHTBTDBTI2QRUV3ZAKOHJQOTIA, seg_pos={0 0}, upload=485040]: Long tail cancellation due to optimal threshold Dec 28 16:42:47.627569559 (553h13m47.498652604s) [noise=true , address=45.91.116.11:4119 , piece_id=WJFBON3R2X5IZABBTUYEEHQPUKR5P3QDAXT26MF6Q76X36TFISNA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 16:42:47.627575330 (553h13m47.498658365s) [noise=true , address=45.91.116.11:4119 , piece_id=WJFBON3R2X5IZABBTUYEEHQPUKR5P3QDAXT26MF6Q76X36TFISNA, seg_pos={0 0}, upload=485040]: Long tail cancellation due to optimal threshold Dec 28 16:42:47.627592653 (553h13m47.498675688s) [seg_pos={0 0}, upload=485040]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:42:47.627696591 (553h13m47.498779626s) [upload=485040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:42:47.627701430 (553h13m47.498784465s) [upload=485040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:42:47.927535305 (553h13m47.79861834s) [upload=485040]: Done uploading segments... Dec 28 16:42:47.927553399 (553h13m47.798636434s) [upload=485040]: Upload finished. err: <nil> Duration: 5.929059003s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [X.X.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X..] duration: 3.916286973s trace: https://jaeger.dev.storj.io/trace/c534c57f90f8f28 FAIL (2025-12-28 16:43:30 UTC) orbiter warning: context deadline exceeded Duration: 3.91632757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.X] duration: 30.003646363s trace: https://jaeger.dev.storj.io/trace/519a26b1f03b2094 FAIL (2025-12-28 16:43:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 16:42:45.032946719 (553h13m44.904029764s) [upload=485040]: Starting upload Dec 28 16:42:45.033058151 (553h13m44.904141196s) [upload=485040]: Done starting upload Dec 28 16:42:45.033190033 (553h13m44.904273078s) [upload=485040]: Uploading segments... Dec 28 16:42:45.033448554 (553h13m44.9045316s) [upload=485040]: Got next segment. Inline: false Dec 28 16:42:45.033482599 (553h13m44.904565644s) [upload=485040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:42:45.033488040 (553h13m44.904571075s) [upload=485040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:42:45.276784437 (553h13m45.147867482s) [seg_pos={0 0}, upload=485040]: Begin upload segment... Dec 28 16:42:45.276839502 (553h13m45.147922537s) [seg_pos={0 0}, upload=485040]: Done begin upload segment. Dec 28 16:42:45.276877514 (553h13m45.147960559s) [noise=true , address=145.79.227.14:4840 , piece_id=5ZNNFOAOCO3VGTOTA6AB4MUMMDGS37XIBEC4C3KVCTVI2FJGZ3XQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.276926738 (553h13m45.148009773s) [noise=true , address=145.79.227.38:4977 , piece_id=J4SWU3AIQBLTZQH7232LNKGNONJSAKKKLWR7KRSZDF7SCJVLZL6A, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.277413815 (553h13m45.14849686s) [noise=true , address=45.91.116.11:4119 , piece_id=WJFBON3R2X5IZABBTUYEEHQPUKR5P3QDAXT26MF6Q76X36TFISNA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278080174 (553h13m45.149163209s) [noise=true , address=77.90.31.17:4773 , piece_id=F47V7WJ5EECOSNENXTFDLNYX7YWEP732SGXCNE5NHHGFLE526APQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278441913 (553h13m45.149524948s) [noise=true , address=145.79.227.37:4567 , piece_id=ZSZBID2D234S4WRH53PHB5RBMROBWRBLG3VTEQ7RVFD4SUM4X4KQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278577892 (553h13m45.149660937s) [noise=true , address=45.91.116.7:4012 , piece_id=QGW5EMYKZJ2GCG6ZG53TXFUGXYOFLQFCGXU5QTUU7CKMVASEAEUA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.278750060 (553h13m45.149833095s) [noise=true , address=45.91.116.5:4299 , piece_id=DBJ4S3XAAZT7TUI3DBH53GXOIOLXM3MM37BOCKHGOKCVBWMYIVOA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279069097 (553h13m45.150152143s) [noise=true , address=145.79.227.11:4731 , piece_id=EQ5BDZZL65OO7VJQWC357TK6DK53423FRLEYO3TDS3O7XIQIGPHA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279402301 (553h13m45.150485346s) [noise=true , address=77.90.31.18:4805 , piece_id=CY5HMDT2ECKE5O7JKFETTNWOADX3GTKBXJPDSHXX3R2U3AKDF57A, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279672576 (553h13m45.150755621s) [noise=true , address=45.91.116.14:4190 , piece_id=VWPU4XX4VRORTASUQ6EJ2YCW2L5SKMFAX4FGZMUBVX2FYOBXCORQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.279960294 (553h13m45.151043339s) [noise=true , address=145.79.227.47:4266 , piece_id=M4FZPADV4BFVZ5Z7NPN7QGJUNYEB2DJMSDALH6DAS3PXTNHFVJUQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.280260987 (553h13m45.151344022s) [noise=true , address=145.79.227.42:4123 , piece_id=BV7DW22FZIW4FBZDUHPYJ4EOARTF2U57HHQ47AUIWQOKQCO2N3XA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.280635951 (553h13m45.151718986s) [noise=true , address=145.79.227.45:4231 , piece_id=6RCTC4GLP42DI4WSC2MFRNMVGXNQ657XOV2DFMGLWHP6U7RD42PQ, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.280964486 (553h13m45.152047521s) [noise=true , address=45.91.116.13:4156 , piece_id=RETU4J64HWVJKSEE5DJK3LMYKGPEJG4MJR4GDBCR5CQKK7LGMZ2A, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.281145160 (553h13m45.152228205s) [noise=true , address=145.79.227.43:4158 , piece_id=VK3AWLWHVHGLIWFX5BCHIF7QSFMHTBTDBTI2QRUV3ZAKOHJQOTIA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:45.281829383 (553h13m45.152912428s) [noise=true , address=145.79.227.16:4911 , piece_id=HJFD6SDEX4S7MYFTMXKNS7DR2SBMBVWR432AUYCDUHLSWJAFCAIA, seg_pos={0 0}, upload=485040]: Uploading piece... Dec 28 16:42:46.900749110 (553h13m46.771832145s) [upload=485040]: Next returned nil segment Dec 28 16:42:46.900763998 (553h13m46.771847034s) [upload=485040]: Waiting for error group managing segments... Dec 28 16:42:47.280334636 (553h13m47.151417671s) [noise=true , address=145.79.227.45:4231 , piece_id=6RCTC4GLP42DI4WSC2MFRNMVGXNQ657XOV2DFMGLWHP6U7RD42PQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.282940407 (553h13m47.154023452s) [noise=true , address=145.79.227.11:4731 , piece_id=EQ5BDZZL65OO7VJQWC357TK6DK53423FRLEYO3TDS3O7XIQIGPHA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.295013496 (553h13m47.166096531s) [noise=true , address=145.79.227.42:4123 , piece_id=BV7DW22FZIW4FBZDUHPYJ4EOARTF2U57HHQ47AUIWQOKQCO2N3XA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.299210378 (553h13m47.170293413s) [noise=true , address=145.79.227.14:4840 , piece_id=5ZNNFOAOCO3VGTOTA6AB4MUMMDGS37XIBEC4C3KVCTVI2FJGZ3XQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.302133544 (553h13m47.173216589s) [noise=true , address=145.79.227.16:4911 , piece_id=HJFD6SDEX4S7MYFTMXKNS7DR2SBMBVWR432AUYCDUHLSWJAFCAIA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.309764956 (553h13m47.180847991s) [noise=true , address=145.79.227.47:4266 , piece_id=M4FZPADV4BFVZ5Z7NPN7QGJUNYEB2DJMSDALH6DAS3PXTNHFVJUQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.313747769 (553h13m47.184830804s) [noise=true , address=145.79.227.37:4567 , piece_id=ZSZBID2D234S4WRH53PHB5RBMROBWRBLG3VTEQ7RVFD4SUM4X4KQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.324597229 (553h13m47.195680264s) [noise=true , address=145.79.227.38:4977 , piece_id=J4SWU3AIQBLTZQH7232LNKGNONJSAKKKLWR7KRSZDF7SCJVLZL6A, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.470504010 (553h13m47.341587095s) [noise=true , address=77.90.31.17:4773 , piece_id=F47V7WJ5EECOSNENXTFDLNYX7YWEP732SGXCNE5NHHGFLE526APQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.529062365 (553h13m47.4001454s) [noise=true , address=45.91.116.5:4299 , piece_id=DBJ4S3XAAZT7TUI3DBH53GXOIOLXM3MM37BOCKHGOKCVBWMYIVOA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.594172245 (553h13m47.46525528s) [noise=true , address=77.90.31.18:4805 , piece_id=CY5HMDT2ECKE5O7JKFETTNWOADX3GTKBXJPDSHXX3R2U3AKDF57A, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.611947371 (553h13m47.483030406s) [noise=true , address=45.91.116.7:4012 , piece_id=QGW5EMYKZJ2GCG6ZG53TXFUGXYOFLQFCGXU5QTUU7CKMVASEAEUA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.626966491 (553h13m47.498049536s) [noise=true , address=45.91.116.13:4156 , piece_id=RETU4J64HWVJKSEE5DJK3LMYKGPEJG4MJR4GDBCR5CQKK7LGMZ2A, seg_pos={0 0}, upload=485040]: Done uploading piece. err: <nil> Dec 28 16:42:47.626986168 (553h13m47.498069203s) [seg_pos={0 0}, upload=485040]: Segment reached optimal threshold of 13 pieces. Dec 28 16:42:47.627236225 (553h13m47.49831927s) [noise=true , address=45.91.116.14:4190 , piece_id=VWPU4XX4VRORTASUQ6EJ2YCW2L5SKMFAX4FGZMUBVX2FYOBXCORQ, seg_pos={0 0}, upload=485040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:42102->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:42102->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42102->45.91.116.14:4190: use of closed network connection Dec 28 16:42:47.627245873 (553h13m47.498328908s) [noise=true , address=45.91.116.14:4190 , piece_id=VWPU4XX4VRORTASUQ6EJ2YCW2L5SKMFAX4FGZMUBVX2FYOBXCORQ, seg_pos={0 0}, upload=485040]: Long tail cancellation due to optimal threshold Dec 28 16:42:47.627374739 (553h13m47.498457784s) [noise=true , address=145.79.227.43:4158 , piece_id=VK3AWLWHVHGLIWFX5BCHIF7QSFMHTBTDBTI2QRUV3ZAKOHJQOTIA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 16:42:47.627403462 (553h13m47.498486508s) [noise=true , address=145.79.227.43:4158 , piece_id=VK3AWLWHVHGLIWFX5BCHIF7QSFMHTBTDBTI2QRUV3ZAKOHJQOTIA, seg_pos={0 0}, upload=485040]: Long tail cancellation due to optimal threshold Dec 28 16:42:47.627569559 (553h13m47.498652604s) [noise=true , address=45.91.116.11:4119 , piece_id=WJFBON3R2X5IZABBTUYEEHQPUKR5P3QDAXT26MF6Q76X36TFISNA, seg_pos={0 0}, upload=485040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 16:42:47.627575330 (553h13m47.498658365s) [noise=true , address=45.91.116.11:4119 , piece_id=WJFBON3R2X5IZABBTUYEEHQPUKR5P3QDAXT26MF6Q76X36TFISNA, seg_pos={0 0}, upload=485040]: Long tail cancellation due to optimal threshold Dec 28 16:42:47.627592653 (553h13m47.498675688s) [seg_pos={0 0}, upload=485040]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:42:47.627696591 (553h13m47.498779626s) [upload=485040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:42:47.627701430 (553h13m47.498784465s) [upload=485040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:42:47.927535305 (553h13m47.79861834s) [upload=485040]: Done uploading segments... Dec 28 16:42:47.927553399 (553h13m47.798636434s) [upload=485040]: Upload finished. err: <nil> Duration: 30.00367118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.............X.........................X.........] duration: 15.349945991s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:42:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 15.349976901s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.X.X.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X.] duration: 3.578706121s trace: https://jaeger.dev.storj.io/trace/c534c57f90f8f28 FAIL (2025-12-28 16:42:33 UTC) orbiter warning: context deadline exceeded Duration: 3.578728986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.............X.........................X........] duration: 32.191990366s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:42:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 32.192005073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.......X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X.] duration: 6.02357231s trace: https://jaeger.dev.storj.io/trace/2eb9ace201de0e23 FAIL (2025-12-28 16:42:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: SWV4Y3MKOAEOP64UV5JIACJY62HO6VOTNUEZIG77KU53UGB4DJTA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 6ZTSD4JRJDDNNUUPRMCEWWB6FPGB2YEFH4QXLCQRM2253OITVEYA) context deadline exceeded ======LOGS====== Dec 28 16:41:21.960379574 (553h12m21.831462609s) [upload=485018]: Starting upload Dec 28 16:41:21.960475687 (553h12m21.831558732s) [upload=485018]: Done starting upload Dec 28 16:41:21.960570207 (553h12m21.831653253s) [upload=485018]: Uploading segments... Dec 28 16:41:21.960709964 (553h12m21.831792999s) [upload=485018]: Got next segment. Inline: false Dec 28 16:41:21.960757884 (553h12m21.83184095s) [upload=485018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:41:21.960763755 (553h12m21.831846791s) [upload=485018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:41:22.220387453 (553h12m22.091470498s) [seg_pos={0 0}, upload=485018]: Begin upload segment... Dec 28 16:41:22.220439392 (553h12m22.091522427s) [seg_pos={0 0}, upload=485018]: Done begin upload segment. Dec 28 16:41:22.220472736 (553h12m22.091555781s) [noise=true , address=77.90.31.21:4878 , piece_id=IJBPSMO4H4XBMWGNHTGBDJM2E3HEFKQCMZNCZTBMIOBPSQI3UZZQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.220537349 (553h12m22.091620394s) [noise=true , address=145.79.227.47:4265 , piece_id=SKUYULNBAVRPCOGBBH4L2JFU56PF7H7FZSY7KJHU32DYPJPQ5PDA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.221357269 (553h12m22.092440315s) [noise=true , address=145.79.227.11:4732 , piece_id=W2CB2TIXNIGICT42FKJPQI2YVKCQUPU5YFTHU7WJB3BIYAUUQPWQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.221785926 (553h12m22.092868971s) [noise=true , address=145.79.227.41:4085 , piece_id=PMXQLUYTULZEMO4SE7EN2BWEXWCK5PTQIYGIUP7ETM4NOFR6I4UQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.222586912 (553h12m22.093669947s) [noise=true , address=145.79.227.27:4690 , piece_id=JVSOQGEFDZUBIY7FPG3QD6C2JBWRI6T7PZWPUFYVL6EHOEMXQWKQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.223393878 (553h12m22.094476913s) [noise=true , address=145.79.227.13:4804 , piece_id=Q5FW5CRGZ6FN73Q6BWZNVBM56CWC7TMT54S6CHXLUCHGSFUALVCA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.225138670 (553h12m22.096221705s) [noise=true , address=145.79.227.39:4015 , piece_id=OHHQYBHQOUIRGMKZ6DORPAD2BHH4O65W5VGF5GTSCO3PNF33G7PQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.226375786 (553h12m22.097458822s) [noise=true , address=77.90.31.10:4520 , piece_id=SWV4Y3MKOAEOP64UV5JIACJY62HO6VOTNUEZIG77KU53UGB4DJTA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.227693465 (553h12m22.0987765s) [noise=true , address=145.79.227.52:4447 , piece_id=MDOFVIUSQ4B43VG66AGCSCY73JTVJLOPMYGYUZZGGDB72HAGSEYQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.229122105 (553h12m22.10020515s) [noise=true , address=45.91.116.5:4300 , piece_id=TFK4FS4G5G2QZHHFXACF33BITZON53TZTAVPE56LF75IR6RBS3DA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.229719353 (553h12m22.100802388s) [noise=true , address=145.79.227.38:4979 , piece_id=SUP6CVC56P2DUMXPBMTGKU3CG3PGPT555ZWMK4PJL6CZHT5WYFTQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.231347222 (553h12m22.102430267s) [noise=true , address=45.91.116.14:4192 , piece_id=6ZTSD4JRJDDNNUUPRMCEWWB6FPGB2YEFH4QXLCQRM2253OITVEYA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.231939410 (553h12m22.103022445s) [noise=true , address=145.79.227.3:4151 , piece_id=2QNKC6GSMSBQBV2YGU6QRQM6GTSWNMT2YGR4T7KQXWRBBYUP4EQA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.232092311 (553h12m22.103175347s) [noise=true , address=45.91.116.4:4940 , piece_id=I6LNWU6W6DGPIW3SNAEHHQNZXCJWE2BWG7FATLJ6PVPFZONZH3MQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.233638776 (553h12m22.104721821s) [noise=true , address=77.90.31.12:4590 , piece_id=RY2UFOSJO4YTDTODWD4LPR4WHFVGPUJMQHJYTA5XRZWWIEXR4CIA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.234406678 (553h12m22.105489723s) [noise=true , address=145.79.227.48:4302 , piece_id=QEWSGQ6JSTK3WTLQB5YBXAWSDSAGVC37G3EAH3Z3WUZGMJKCVRBQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:23.849610960 (553h12m23.720694005s) [upload=485018]: Next returned nil segment Dec 28 16:41:23.849630106 (553h12m23.720713151s) [upload=485018]: Waiting for error group managing segments... Dec 28 16:41:24.230979170 (553h12m24.102062205s) [noise=true , address=145.79.227.48:4302 , piece_id=QEWSGQ6JSTK3WTLQB5YBXAWSDSAGVC37G3EAH3Z3WUZGMJKCVRBQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.237773567 (553h12m24.108856612s) [noise=true , address=145.79.227.11:4732 , piece_id=W2CB2TIXNIGICT42FKJPQI2YVKCQUPU5YFTHU7WJB3BIYAUUQPWQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.241038615 (553h12m24.11212165s) [noise=true , address=145.79.227.52:4447 , piece_id=MDOFVIUSQ4B43VG66AGCSCY73JTVJLOPMYGYUZZGGDB72HAGSEYQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.242692224 (553h12m24.113775269s) [noise=true , address=145.79.227.3:4151 , piece_id=2QNKC6GSMSBQBV2YGU6QRQM6GTSWNMT2YGR4T7KQXWRBBYUP4EQA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.252284749 (553h12m24.123367784s) [noise=true , address=145.79.227.27:4690 , piece_id=JVSOQGEFDZUBIY7FPG3QD6C2JBWRI6T7PZWPUFYVL6EHOEMXQWKQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.254248870 (553h12m24.125331915s) [noise=true , address=145.79.227.47:4265 , piece_id=SKUYULNBAVRPCOGBBH4L2JFU56PF7H7FZSY7KJHU32DYPJPQ5PDA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.255103847 (553h12m24.126186882s) [noise=true , address=145.79.227.39:4015 , piece_id=OHHQYBHQOUIRGMKZ6DORPAD2BHH4O65W5VGF5GTSCO3PNF33G7PQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.266792485 (553h12m24.13787552s) [noise=true , address=145.79.227.41:4085 , piece_id=PMXQLUYTULZEMO4SE7EN2BWEXWCK5PTQIYGIUP7ETM4NOFR6I4UQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.267903150 (553h12m24.138986195s) [noise=true , address=145.79.227.38:4979 , piece_id=SUP6CVC56P2DUMXPBMTGKU3CG3PGPT555ZWMK4PJL6CZHT5WYFTQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.289332294 (553h12m24.160415329s) [noise=true , address=145.79.227.13:4804 , piece_id=Q5FW5CRGZ6FN73Q6BWZNVBM56CWC7TMT54S6CHXLUCHGSFUALVCA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.621349842 (553h12m24.492432877s) [noise=true , address=45.91.116.4:4940 , piece_id=I6LNWU6W6DGPIW3SNAEHHQNZXCJWE2BWG7FATLJ6PVPFZONZH3MQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.717496642 (553h12m24.588579677s) [noise=true , address=77.90.31.10:4520 , piece_id=SWV4Y3MKOAEOP64UV5JIACJY62HO6VOTNUEZIG77KU53UGB4DJTA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:25.142477787 (553h12m25.013560832s) [noise=true , address=45.91.116.14:4192 , piece_id=6ZTSD4JRJDDNNUUPRMCEWWB6FPGB2YEFH4QXLCQRM2253OITVEYA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:25.142497605 (553h12m25.01358064s) [seg_pos={0 0}, upload=485018]: Segment reached optimal threshold of 13 pieces. Dec 28 16:41:25.142702676 (553h12m25.013785721s) [noise=true , address=77.90.31.12:4590 , piece_id=RY2UFOSJO4YTDTODWD4LPR4WHFVGPUJMQHJYTA5XRZWWIEXR4CIA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 16:41:25.142713646 (553h12m25.013796691s) [noise=true , address=77.90.31.12:4590 , piece_id=RY2UFOSJO4YTDTODWD4LPR4WHFVGPUJMQHJYTA5XRZWWIEXR4CIA, seg_pos={0 0}, upload=485018]: Long tail cancellation due to optimal threshold Dec 28 16:41:25.142876056 (553h12m25.013959101s) [noise=true , address=45.91.116.5:4300 , piece_id=TFK4FS4G5G2QZHHFXACF33BITZON53TZTAVPE56LF75IR6RBS3DA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 16:41:25.142891585 (553h12m25.01397463s) [noise=true , address=45.91.116.5:4300 , piece_id=TFK4FS4G5G2QZHHFXACF33BITZON53TZTAVPE56LF75IR6RBS3DA, seg_pos={0 0}, upload=485018]: Long tail cancellation due to optimal threshold Dec 28 16:41:25.143026051 (553h12m25.014109096s) [noise=true , address=77.90.31.21:4878 , piece_id=IJBPSMO4H4XBMWGNHTGBDJM2E3HEFKQCMZNCZTBMIOBPSQI3UZZQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 16:41:25.143036582 (553h12m25.014119627s) [noise=true , address=77.90.31.21:4878 , piece_id=IJBPSMO4H4XBMWGNHTGBDJM2E3HEFKQCMZNCZTBMIOBPSQI3UZZQ, seg_pos={0 0}, upload=485018]: Long tail cancellation due to optimal threshold Dec 28 16:41:25.143059825 (553h12m25.014142861s) [seg_pos={0 0}, upload=485018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:41:25.143157001 (553h12m25.014240046s) [upload=485018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:41:25.143162812 (553h12m25.014245847s) [upload=485018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:41:25.435603619 (553h12m25.306686664s) [upload=485018]: Done uploading segments... Dec 28 16:41:25.435623898 (553h12m25.306706933s) [upload=485018]: Upload finished. err: <nil> Duration: 6.023600855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.......X....XXXXX.........X..XX..X..XX..XXXX...X...XX....X] duration: 30.005756519s trace: https://jaeger.dev.storj.io/trace/2eb9ace201de0e23 FAIL (2025-12-28 16:41:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: SWV4Y3MKOAEOP64UV5JIACJY62HO6VOTNUEZIG77KU53UGB4DJTA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 6ZTSD4JRJDDNNUUPRMCEWWB6FPGB2YEFH4QXLCQRM2253OITVEYA) context deadline exceeded ======LOGS====== Dec 28 16:41:21.960379574 (553h12m21.831462609s) [upload=485018]: Starting upload Dec 28 16:41:21.960475687 (553h12m21.831558732s) [upload=485018]: Done starting upload Dec 28 16:41:21.960570207 (553h12m21.831653253s) [upload=485018]: Uploading segments... Dec 28 16:41:21.960709964 (553h12m21.831792999s) [upload=485018]: Got next segment. Inline: false Dec 28 16:41:21.960757884 (553h12m21.83184095s) [upload=485018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:41:21.960763755 (553h12m21.831846791s) [upload=485018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:41:22.220387453 (553h12m22.091470498s) [seg_pos={0 0}, upload=485018]: Begin upload segment... Dec 28 16:41:22.220439392 (553h12m22.091522427s) [seg_pos={0 0}, upload=485018]: Done begin upload segment. Dec 28 16:41:22.220472736 (553h12m22.091555781s) [noise=true , address=77.90.31.21:4878 , piece_id=IJBPSMO4H4XBMWGNHTGBDJM2E3HEFKQCMZNCZTBMIOBPSQI3UZZQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.220537349 (553h12m22.091620394s) [noise=true , address=145.79.227.47:4265 , piece_id=SKUYULNBAVRPCOGBBH4L2JFU56PF7H7FZSY7KJHU32DYPJPQ5PDA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.221357269 (553h12m22.092440315s) [noise=true , address=145.79.227.11:4732 , piece_id=W2CB2TIXNIGICT42FKJPQI2YVKCQUPU5YFTHU7WJB3BIYAUUQPWQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.221785926 (553h12m22.092868971s) [noise=true , address=145.79.227.41:4085 , piece_id=PMXQLUYTULZEMO4SE7EN2BWEXWCK5PTQIYGIUP7ETM4NOFR6I4UQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.222586912 (553h12m22.093669947s) [noise=true , address=145.79.227.27:4690 , piece_id=JVSOQGEFDZUBIY7FPG3QD6C2JBWRI6T7PZWPUFYVL6EHOEMXQWKQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.223393878 (553h12m22.094476913s) [noise=true , address=145.79.227.13:4804 , piece_id=Q5FW5CRGZ6FN73Q6BWZNVBM56CWC7TMT54S6CHXLUCHGSFUALVCA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.225138670 (553h12m22.096221705s) [noise=true , address=145.79.227.39:4015 , piece_id=OHHQYBHQOUIRGMKZ6DORPAD2BHH4O65W5VGF5GTSCO3PNF33G7PQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.226375786 (553h12m22.097458822s) [noise=true , address=77.90.31.10:4520 , piece_id=SWV4Y3MKOAEOP64UV5JIACJY62HO6VOTNUEZIG77KU53UGB4DJTA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.227693465 (553h12m22.0987765s) [noise=true , address=145.79.227.52:4447 , piece_id=MDOFVIUSQ4B43VG66AGCSCY73JTVJLOPMYGYUZZGGDB72HAGSEYQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.229122105 (553h12m22.10020515s) [noise=true , address=45.91.116.5:4300 , piece_id=TFK4FS4G5G2QZHHFXACF33BITZON53TZTAVPE56LF75IR6RBS3DA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.229719353 (553h12m22.100802388s) [noise=true , address=145.79.227.38:4979 , piece_id=SUP6CVC56P2DUMXPBMTGKU3CG3PGPT555ZWMK4PJL6CZHT5WYFTQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.231347222 (553h12m22.102430267s) [noise=true , address=45.91.116.14:4192 , piece_id=6ZTSD4JRJDDNNUUPRMCEWWB6FPGB2YEFH4QXLCQRM2253OITVEYA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.231939410 (553h12m22.103022445s) [noise=true , address=145.79.227.3:4151 , piece_id=2QNKC6GSMSBQBV2YGU6QRQM6GTSWNMT2YGR4T7KQXWRBBYUP4EQA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.232092311 (553h12m22.103175347s) [noise=true , address=45.91.116.4:4940 , piece_id=I6LNWU6W6DGPIW3SNAEHHQNZXCJWE2BWG7FATLJ6PVPFZONZH3MQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.233638776 (553h12m22.104721821s) [noise=true , address=77.90.31.12:4590 , piece_id=RY2UFOSJO4YTDTODWD4LPR4WHFVGPUJMQHJYTA5XRZWWIEXR4CIA, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:22.234406678 (553h12m22.105489723s) [noise=true , address=145.79.227.48:4302 , piece_id=QEWSGQ6JSTK3WTLQB5YBXAWSDSAGVC37G3EAH3Z3WUZGMJKCVRBQ, seg_pos={0 0}, upload=485018]: Uploading piece... Dec 28 16:41:23.849610960 (553h12m23.720694005s) [upload=485018]: Next returned nil segment Dec 28 16:41:23.849630106 (553h12m23.720713151s) [upload=485018]: Waiting for error group managing segments... Dec 28 16:41:24.230979170 (553h12m24.102062205s) [noise=true , address=145.79.227.48:4302 , piece_id=QEWSGQ6JSTK3WTLQB5YBXAWSDSAGVC37G3EAH3Z3WUZGMJKCVRBQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.237773567 (553h12m24.108856612s) [noise=true , address=145.79.227.11:4732 , piece_id=W2CB2TIXNIGICT42FKJPQI2YVKCQUPU5YFTHU7WJB3BIYAUUQPWQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.241038615 (553h12m24.11212165s) [noise=true , address=145.79.227.52:4447 , piece_id=MDOFVIUSQ4B43VG66AGCSCY73JTVJLOPMYGYUZZGGDB72HAGSEYQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.242692224 (553h12m24.113775269s) [noise=true , address=145.79.227.3:4151 , piece_id=2QNKC6GSMSBQBV2YGU6QRQM6GTSWNMT2YGR4T7KQXWRBBYUP4EQA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.252284749 (553h12m24.123367784s) [noise=true , address=145.79.227.27:4690 , piece_id=JVSOQGEFDZUBIY7FPG3QD6C2JBWRI6T7PZWPUFYVL6EHOEMXQWKQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.254248870 (553h12m24.125331915s) [noise=true , address=145.79.227.47:4265 , piece_id=SKUYULNBAVRPCOGBBH4L2JFU56PF7H7FZSY7KJHU32DYPJPQ5PDA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.255103847 (553h12m24.126186882s) [noise=true , address=145.79.227.39:4015 , piece_id=OHHQYBHQOUIRGMKZ6DORPAD2BHH4O65W5VGF5GTSCO3PNF33G7PQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.266792485 (553h12m24.13787552s) [noise=true , address=145.79.227.41:4085 , piece_id=PMXQLUYTULZEMO4SE7EN2BWEXWCK5PTQIYGIUP7ETM4NOFR6I4UQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.267903150 (553h12m24.138986195s) [noise=true , address=145.79.227.38:4979 , piece_id=SUP6CVC56P2DUMXPBMTGKU3CG3PGPT555ZWMK4PJL6CZHT5WYFTQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.289332294 (553h12m24.160415329s) [noise=true , address=145.79.227.13:4804 , piece_id=Q5FW5CRGZ6FN73Q6BWZNVBM56CWC7TMT54S6CHXLUCHGSFUALVCA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.621349842 (553h12m24.492432877s) [noise=true , address=45.91.116.4:4940 , piece_id=I6LNWU6W6DGPIW3SNAEHHQNZXCJWE2BWG7FATLJ6PVPFZONZH3MQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:24.717496642 (553h12m24.588579677s) [noise=true , address=77.90.31.10:4520 , piece_id=SWV4Y3MKOAEOP64UV5JIACJY62HO6VOTNUEZIG77KU53UGB4DJTA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:25.142477787 (553h12m25.013560832s) [noise=true , address=45.91.116.14:4192 , piece_id=6ZTSD4JRJDDNNUUPRMCEWWB6FPGB2YEFH4QXLCQRM2253OITVEYA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: <nil> Dec 28 16:41:25.142497605 (553h12m25.01358064s) [seg_pos={0 0}, upload=485018]: Segment reached optimal threshold of 13 pieces. Dec 28 16:41:25.142702676 (553h12m25.013785721s) [noise=true , address=77.90.31.12:4590 , piece_id=RY2UFOSJO4YTDTODWD4LPR4WHFVGPUJMQHJYTA5XRZWWIEXR4CIA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 16:41:25.142713646 (553h12m25.013796691s) [noise=true , address=77.90.31.12:4590 , piece_id=RY2UFOSJO4YTDTODWD4LPR4WHFVGPUJMQHJYTA5XRZWWIEXR4CIA, seg_pos={0 0}, upload=485018]: Long tail cancellation due to optimal threshold Dec 28 16:41:25.142876056 (553h12m25.013959101s) [noise=true , address=45.91.116.5:4300 , piece_id=TFK4FS4G5G2QZHHFXACF33BITZON53TZTAVPE56LF75IR6RBS3DA, seg_pos={0 0}, upload=485018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 16:41:25.142891585 (553h12m25.01397463s) [noise=true , address=45.91.116.5:4300 , piece_id=TFK4FS4G5G2QZHHFXACF33BITZON53TZTAVPE56LF75IR6RBS3DA, seg_pos={0 0}, upload=485018]: Long tail cancellation due to optimal threshold Dec 28 16:41:25.143026051 (553h12m25.014109096s) [noise=true , address=77.90.31.21:4878 , piece_id=IJBPSMO4H4XBMWGNHTGBDJM2E3HEFKQCMZNCZTBMIOBPSQI3UZZQ, seg_pos={0 0}, upload=485018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 16:41:25.143036582 (553h12m25.014119627s) [noise=true , address=77.90.31.21:4878 , piece_id=IJBPSMO4H4XBMWGNHTGBDJM2E3HEFKQCMZNCZTBMIOBPSQI3UZZQ, seg_pos={0 0}, upload=485018]: Long tail cancellation due to optimal threshold Dec 28 16:41:25.143059825 (553h12m25.014142861s) [seg_pos={0 0}, upload=485018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:41:25.143157001 (553h12m25.014240046s) [upload=485018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:41:25.143162812 (553h12m25.014245847s) [upload=485018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:41:25.435603619 (553h12m25.306686664s) [upload=485018]: Done uploading segments... Dec 28 16:41:25.435623898 (553h12m25.306706933s) [upload=485018]: Upload finished. err: <nil> Duration: 30.005823759s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [..X.X.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.X] duration: 5.001292668s trace: https://jaeger.dev.storj.io/trace/c534c57f90f8f28 FAIL (2025-12-28 16:41:47 UTC) orbiter warning: context deadline exceeded Duration: 5.001315671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.............X.........................X.......] duration: 36.658719453s trace: https://jaeger.dev.storj.io/trace/260910777da62590 FAIL (2025-12-28 16:41:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oad=484920]: Uploading piece... Dec 28 16:34:29.213041513 (553h5m29.084124548s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.213862947 (553h5m29.084945982s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214251237 (553h5m29.085334282s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.214748433 (553h5m29.085831468s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215054536 (553h5m29.086137571s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215291878 (553h5m29.086374913s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215341462 (553h5m29.086424507s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.215701177 (553h5m29.086784212s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216132429 (553h5m29.087215464s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216170732 (553h5m29.087253767s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216539213 (553h5m29.087622258s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216757389 (553h5m29.087840434s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.216889079 (553h5m29.087972124s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217322054 (553h5m29.088405089s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.217894013 (553h5m29.088977048s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.218662205 (553h5m29.08974525s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:29.219363471 (553h5m29.090446517s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Uploading piece... Dec 28 16:34:30.540650713 (553h5m30.411733758s) [upload=484920]: Next returned nil segment Dec 28 16:34:30.540668627 (553h5m30.411751682s) [upload=484920]: Waiting for error group managing segments... Dec 28 16:34:30.886938072 (553h5m30.758021107s) [noise=true , address=64.49.13.155:4921 , piece_id=4PCKVX4Z2PEM7VJKQ4EEON54TTIPVRBTVCVTTV2GQQZQJNZNQ47Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.906289781 (553h5m30.777372827s) [noise=true , address=64.49.13.137:4494 , piece_id=BDYDJPNPEKDH3SXDQ7GI2VMJTLAT2GE753NF6DFSQIAMHWP6THFA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:30.979664449 (553h5m30.850747484s) [noise=true , address=64.49.13.26:4951 , piece_id=TMMUQITGTMXBO47ZGYU5UNGZ6P3EVWJB42YUACBS5TDHHIKRXJBQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.081604941 (553h5m30.952687976s) [noise=true , address=145.79.227.31:4804 , piece_id=VXSCCV7LVN7KOYEC3N6GEJ4TU2S7KTH6PFAHUE6FYBL4AXIIKLLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.255064967 (553h5m31.126148013s) [noise=true , address=145.79.227.13:4824 , piece_id=7YRYCJF2APQLP774VDKQNKV2AJYQMDCB6ZPTFJSR7ZN6LLBSWXUQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.290570366 (553h5m31.161653421s) [noise=true , address=77.90.31.17:4803 , piece_id=ZUWDWDNQITY5WIXYFPCMLRDVIPXIS4JLINBCZEXHT3KFZFJJMS6A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.369098606 (553h5m31.240181651s) [noise=true , address=77.90.31.18:4823 , piece_id=YTOXIGCG6FVF5H4F4NHPKH37XVI6AYSB7WQPNQPXORTNYWHDAO2Q, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.499720722 (553h5m31.370803757s) [noise=true , address=45.91.116.9:4276 , piece_id=SIKYG6OLIHYCL3TNXEOT5WEFI5D6TFMQH6626F7NMYYUW5KV5GNQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.680584587 (553h5m31.551667633s) [noise=true , address=77.90.31.14:4673 , piece_id=SOT7TR33RYLWCEV75DFVM4LPPZHLS7H2ZCDDQMY4IAJM7HGO3QHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:31.999402671 (553h5m31.870485716s) [noise=true , address=145.79.227.44:4225 , piece_id=K3OQDUQREYJAOC67Y5TKZQ4EPY7UVY66E2EMYTPJKYTVGD6OOK3A, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.012854264 (553h5m31.883937309s) [noise=true , address=145.79.227.12:4770 , piece_id=DTM2KWXXPVGGNBGUUP2K5E4YKR4BU476P5O3N46CABIYGSHEOUKQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.029204538 (553h5m31.900287583s) [noise=true , address=145.79.227.49:4357 , piece_id=MM2E6QQY45QR34FXLENPF6Y65G4Z7LLBEFALCMHU32H7FLEINVHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.040963730 (553h5m31.912046765s) [noise=true , address=145.79.227.30:4785 , piece_id=JWXAJS5SO7BEH6TNYHIYALDNZQPX5WHFE7F6LVL5LKWKSLT6KTWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064445944 (553h5m31.935528989s) [noise=true , address=145.79.227.43:4161 , piece_id=D4APN7XMIDLEYJVDBDWPH4L7DXXYILBSISUPV4FS35PEQVLB6SSA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: <nil> Dec 28 16:34:32.064473987 (553h5m31.935557032s) [seg_pos={0 0}, upload=484920]: Segment reached optimal threshold of 14 pieces. Dec 28 16:34:32.064671543 (553h5m31.935754578s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064678596 (553h5m31.935761631s) [noise=true , address=145.79.227.33:4904 , piece_id=2AJB3BRUL46A3P5FU5TCC5OJQHQ4QMROH3X742OBACMGKRUP6LHA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064788907 (553h5m31.935871942s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064803454 (553h5m31.935886499s) [noise=true , address=145.79.227.40:4063 , piece_id=HIF6DQ6MTTINUIGZLIPBUIAWVGK662IMZK4HWMAKO2CXTNJTN7GA, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064813022 (553h5m31.935896067s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18KKE2MHrYNf3WJ2TvNXca71uHbjLTwsHu36aMXbGpMt73FyUk): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064817671 (553h5m31.935900706s) [noise=true , address=64.49.13.170:4727 , piece_id=O647FL47LZUTFXI362WTC3GYPOZAHE2IUA5QWPRVLQKXIQI7UBWQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064898195 (553h5m31.93598123s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TFMaFbwx2ousAeP1iPryTHHoWBHgjPjXiBGx6bSLyCgRHz51A): protocol: expected piece hash; context canceled; EOF Dec 28 16:34:32.064905128 (553h5m31.935988163s) [noise=true , address=145.79.227.34:4922 , piece_id=EL4S3JA2XRIHAJSGEF4F2WMAKI5UXCJLD6JWPK5NYX5ZLCIFISLQ, seg_pos={0 0}, upload=484920]: Long tail cancellation due to optimal threshold Dec 28 16:34:32.064945324 (553h5m31.936028359s) [seg_pos={0 0}, upload=484920]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:34:32.065031028 (553h5m31.936114073s) [upload=484920]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:34:32.065038782 (553h5m31.936121828s) [upload=484920]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:34:32.371516498 (553h5m32.242599543s) [upload=484920]: Done uploading segments... Dec 28 16:34:32.371538690 (553h5m32.242621736s) [upload=484920]: Upload finished. err: <nil> Duration: 36.658744622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [...X.X.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X.] duration: 3.461037281s trace: https://jaeger.dev.storj.io/trace/53434135046f83d4 FAIL (2025-12-28 16:41:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46106304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [X...X.X.XXXX..XXXXX.X..XXX.X.XXXX..XX.X...X..XXX.XX.X.XXXX.X] duration: 5.001382299s trace: https://jaeger.dev.storj.io/trace/53434135046f83d4 FAIL (2025-12-28 16:40:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00140424s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)