Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.914034097s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:29:40 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 4.914064054s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX.] duration: 4.087911348s trace: https://jaeger.dev.storj.io/trace/2dc6321c59798a09 FAIL (2025-12-28 05:29:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.088084867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............................X.........] duration: 47.014643785s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:28:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 47.014661357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.X] duration: 30.002941327s trace: https://jaeger.dev.storj.io/trace/23ef51535393e441 FAIL (2025-12-28 05:28:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4DPOSDFZ5WR74UVQJ7ZPZ3BLMOWUUYNZSX4CPSJSAAUVUZF2CDTA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OO5HUSPIQJA3HIAURZLPXMS5C52JDFHXNV7Y7WEAMDD6KJIXV2FQ) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: D5TCX7Z46FOHOPNHXJ55WMER7EAFJYNM66ADZYWALINWG7AV2CHA) context deadline exceeded ======LOGS====== Dec 28 05:28:13.352186486 (541h59m13.223269521s) [upload=475285]: Starting upload Dec 28 05:28:13.352258252 (541h59m13.223341298s) [upload=475285]: Done starting upload Dec 28 05:28:13.352465868 (541h59m13.223548913s) [upload=475285]: Uploading segments... Dec 28 05:28:13.352530531 (541h59m13.223613576s) [upload=475285]: Got next segment. Inline: false Dec 28 05:28:13.352565888 (541h59m13.223648923s) [upload=475285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:28:13.352571920 (541h59m13.223654965s) [upload=475285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:28:13.598112769 (541h59m13.469195804s) [seg_pos={0 0}, upload=475285]: Begin upload segment... Dec 28 05:28:13.598157113 (541h59m13.469240159s) [seg_pos={0 0}, upload=475285]: Done begin upload segment. Dec 28 05:28:13.598181741 (541h59m13.469264776s) [noise=true , address=45.91.116.11:4119 , piece_id=4DPOSDFZ5WR74UVQJ7ZPZ3BLMOWUUYNZSX4CPSJSAAUVUZF2CDTA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.598242597 (541h59m13.469325662s) [noise=true , address=145.79.227.4:4187 , piece_id=FZ4YLRSSECL5UBWL43EKBWY4VVHLNZ3JZJUAPZNKOO76ZJILOWCA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.598834414 (541h59m13.469917449s) [noise=true , address=145.79.227.52:4447 , piece_id=EKYHXZGATDZ2JIR3RBWTP6AEFONQIFGC36STCIXDSCUX26HBYUEQ, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.599293377 (541h59m13.470376423s) [noise=true , address=145.79.227.47:4267 , piece_id=WTYTQZ2XKS5SYBS2Z47FS72UJ3H3CIGBBXFWWNPMMGP2M7KZD5BA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.599720110 (541h59m13.470803145s) [noise=true , address=77.90.31.11:4554 , piece_id=OS3MWJQFHCZAXO6QAHV7PTP3B245HHUKMLQWLSUDPQS7PKHCGWJA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.600151342 (541h59m13.471234387s) [noise=true , address=145.79.227.37:4566 , piece_id=JFTFUWDCJR3UPZEFXUM77TJPQZ544F7OXUMCXW6SUBURCZV2277Q, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.600584967 (541h59m13.471668002s) [noise=true , address=145.79.227.35:4483 , piece_id=5JEEKTLUF2XC4UKJ3JASBS5EKPOMWRETKDN4S3IRJDCEYHZMAX6Q, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.600597049 (541h59m13.471680125s) [noise=true , address=45.91.116.7:4011 , piece_id=D5TCX7Z46FOHOPNHXJ55WMER7EAFJYNM66ADZYWALINWG7AV2CHA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.600928531 (541h59m13.472011576s) [noise=true , address=45.91.116.9:4264 , piece_id=O5UP6ZOMWR6IOCKSZ4IBANMS6FZRYNPNKIPM3FW7HE3MDOKWKRAQ, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.601119775 (541h59m13.47220282s) [noise=true , address=145.79.227.48:4301 , piece_id=2FBYRE3DG7IDVLLLKH3VW2CBTDCRDFLQUVAMYY5HBVUYI6OFTRBQ, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.601260394 (541h59m13.472343429s) [noise=true , address=77.90.31.15:4700 , piece_id=SWC72WEL3CVSZPI22SS7CNCOFLUV5OA7S4M4FPZZHAHMVXTN7RAQ, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.601576726 (541h59m13.472659771s) [noise=true , address=77.90.31.10:4519 , piece_id=MLL422GMGRHHC7YUESFFP4HTLJM635SEJNG4GOHIJZEX7HOXSBBA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.602039417 (541h59m13.473122452s) [noise=true , address=145.79.227.19:4079 , piece_id=RVIXV5EXETJPAIHYATMCN3OQ2NB4GLJHV5ICKDKBEDW3SVJG24PA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.602416705 (541h59m13.47349974s) [noise=true , address=145.79.227.15:4873 , piece_id=SISLW2KXUGXHRPNK5F2WY4VPM4ADWJIX5KO5POAXNS7WLWWR7OJA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.602821425 (541h59m13.47390446s) [noise=true , address=145.79.227.6:4258 , piece_id=LVHWP45MTMZIGJISDE3VZZ7MAAEAOFJWCDMAPLFXDJIFLRHBAVYA, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:13.604521212 (541h59m13.475604247s) [noise=true , address=45.91.116.14:4190 , piece_id=OO5HUSPIQJA3HIAURZLPXMS5C52JDFHXNV7Y7WEAMDD6KJIXV2FQ, seg_pos={0 0}, upload=475285]: Uploading piece... Dec 28 05:28:15.221333261 (541h59m15.092416306s) [upload=475285]: Next returned nil segment Dec 28 05:28:15.221354531 (541h59m15.092437566s) [upload=475285]: Waiting for error group managing segments... Dec 28 05:28:15.612575860 (541h59m15.483658905s) [noise=true , address=145.79.227.15:4873 , piece_id=SISLW2KXUGXHRPNK5F2WY4VPM4ADWJIX5KO5POAXNS7WLWWR7OJA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.612932639 (541h59m15.484015674s) [noise=true , address=145.79.227.37:4566 , piece_id=JFTFUWDCJR3UPZEFXUM77TJPQZ544F7OXUMCXW6SUBURCZV2277Q, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.617670681 (541h59m15.488753726s) [noise=true , address=145.79.227.47:4267 , piece_id=WTYTQZ2XKS5SYBS2Z47FS72UJ3H3CIGBBXFWWNPMMGP2M7KZD5BA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.622512430 (541h59m15.493595465s) [noise=true , address=145.79.227.6:4258 , piece_id=LVHWP45MTMZIGJISDE3VZZ7MAAEAOFJWCDMAPLFXDJIFLRHBAVYA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.625355373 (541h59m15.496438418s) [noise=true , address=145.79.227.19:4079 , piece_id=RVIXV5EXETJPAIHYATMCN3OQ2NB4GLJHV5ICKDKBEDW3SVJG24PA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.628061867 (541h59m15.499144902s) [noise=true , address=145.79.227.4:4187 , piece_id=FZ4YLRSSECL5UBWL43EKBWY4VVHLNZ3JZJUAPZNKOO76ZJILOWCA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.631325762 (541h59m15.502408807s) [noise=true , address=145.79.227.52:4447 , piece_id=EKYHXZGATDZ2JIR3RBWTP6AEFONQIFGC36STCIXDSCUX26HBYUEQ, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.640433785 (541h59m15.51151683s) [noise=true , address=145.79.227.48:4301 , piece_id=2FBYRE3DG7IDVLLLKH3VW2CBTDCRDFLQUVAMYY5HBVUYI6OFTRBQ, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.863018401 (541h59m15.734101446s) [noise=true , address=45.91.116.14:4190 , piece_id=OO5HUSPIQJA3HIAURZLPXMS5C52JDFHXNV7Y7WEAMDD6KJIXV2FQ, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.872896912 (541h59m15.743979977s) [noise=true , address=45.91.116.7:4011 , piece_id=D5TCX7Z46FOHOPNHXJ55WMER7EAFJYNM66ADZYWALINWG7AV2CHA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:15.889260301 (541h59m15.760343346s) [noise=true , address=45.91.116.11:4119 , piece_id=4DPOSDFZ5WR74UVQJ7ZPZ3BLMOWUUYNZSX4CPSJSAAUVUZF2CDTA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:16.014953270 (541h59m15.886036305s) [noise=true , address=77.90.31.10:4519 , piece_id=MLL422GMGRHHC7YUESFFP4HTLJM635SEJNG4GOHIJZEX7HOXSBBA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:16.529214877 (541h59m16.400297912s) [noise=true , address=77.90.31.11:4554 , piece_id=OS3MWJQFHCZAXO6QAHV7PTP3B245HHUKMLQWLSUDPQS7PKHCGWJA, seg_pos={0 0}, upload=475285]: Done uploading piece. err: <nil> Dec 28 05:28:16.529253901 (541h59m16.400336936s) [seg_pos={0 0}, upload=475285]: Segment reached optimal threshold of 13 pieces. Dec 28 05:28:16.529466936 (541h59m16.400549972s) [noise=true , address=45.91.116.9:4264 , piece_id=O5UP6ZOMWR6IOCKSZ4IBANMS6FZRYNPNKIPM3FW7HE3MDOKWKRAQ, seg_pos={0 0}, upload=475285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 05:28:16.529474340 (541h59m16.400557375s) [noise=true , address=45.91.116.9:4264 , piece_id=O5UP6ZOMWR6IOCKSZ4IBANMS6FZRYNPNKIPM3FW7HE3MDOKWKRAQ, seg_pos={0 0}, upload=475285]: Long tail cancellation due to optimal threshold Dec 28 05:28:16.530124148 (541h59m16.401207183s) [noise=true , address=77.90.31.15:4700 , piece_id=SWC72WEL3CVSZPI22SS7CNCOFLUV5OA7S4M4FPZZHAHMVXTN7RAQ, seg_pos={0 0}, upload=475285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 05:28:16.530135089 (541h59m16.401218124s) [noise=true , address=77.90.31.15:4700 , piece_id=SWC72WEL3CVSZPI22SS7CNCOFLUV5OA7S4M4FPZZHAHMVXTN7RAQ, seg_pos={0 0}, upload=475285]: Long tail cancellation due to optimal threshold Dec 28 05:28:16.530308759 (541h59m16.401391794s) [noise=true , address=145.79.227.35:4483 , piece_id=5JEEKTLUF2XC4UKJ3JASBS5EKPOMWRETKDN4S3IRJDCEYHZMAX6Q, seg_pos={0 0}, upload=475285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 05:28:16.530317617 (541h59m16.401400652s) [noise=true , address=145.79.227.35:4483 , piece_id=5JEEKTLUF2XC4UKJ3JASBS5EKPOMWRETKDN4S3IRJDCEYHZMAX6Q, seg_pos={0 0}, upload=475285]: Long tail cancellation due to optimal threshold Dec 28 05:28:16.530335741 (541h59m16.401418776s) [seg_pos={0 0}, upload=475285]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:28:16.530425582 (541h59m16.401508627s) [upload=475285]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:28:16.530430922 (541h59m16.401513957s) [upload=475285]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:28:16.824000586 (541h59m16.695083631s) [upload=475285]: Done uploading segments... Dec 28 05:28:16.824020313 (541h59m16.695103358s) [upload=475285]: Upload finished. err: <nil> Duration: 30.002969371s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.608119389s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:28:35 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.608147883s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XXX] duration: 5.001267056s trace: https://jaeger.dev.storj.io/trace/2dc6321c59798a09 FAIL (2025-12-28 05:28:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001287394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............................X........] duration: 49.475914374s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:27:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 49.475945385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....XX] duration: 5.000879326s trace: https://jaeger.dev.storj.io/trace/2bbcc6c390590dfd FAIL (2025-12-28 05:27:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000904073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X.] duration: 6.093180741s trace: https://jaeger.dev.storj.io/trace/3b186855697d524c FAIL (2025-12-28 05:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:27:02.906913362 (541h58m2.777996397s) [upload=475268]: Starting upload Dec 28 05:27:02.906987653 (541h58m2.778070698s) [upload=475268]: Done starting upload Dec 28 05:27:02.907112942 (541h58m2.778195997s) [upload=475268]: Uploading segments... Dec 28 05:27:02.907210587 (541h58m2.778293632s) [upload=475268]: Got next segment. Inline: false Dec 28 05:27:02.907331678 (541h58m2.778414723s) [upload=475268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:27:02.907338391 (541h58m2.778421436s) [upload=475268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:27:03.158542814 (541h58m3.029625879s) [seg_pos={0 0}, upload=475268]: Begin upload segment... Dec 28 05:27:03.158766371 (541h58m3.029849416s) [seg_pos={0 0}, upload=475268]: Done begin upload segment. Dec 28 05:27:03.158809653 (541h58m3.029892698s) [noise=true , address=45.91.116.8:4048 , piece_id=XFGIU5DBLMG3CQBAKWUX5SP4RKDSBPLCDBE6I2X525IGLM5I6LWQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.159261533 (541h58m3.030344578s) [noise=true , address=145.79.227.50:4373 , piece_id=7QUOTLRQTUHNDIQXVUHOHBPJI3XM5MOCQN2KQZV4N3ZRLCRWOZZA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.159652347 (541h58m3.030735392s) [noise=true , address=145.79.227.44:4195 , piece_id=TK6S7EW4Q63JFRRLE6UPPREIFZNJNNIFIO4KR2WDLIXNSD6USSSA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.159843091 (541h58m3.030926136s) [noise=true , address=145.79.227.15:4873 , piece_id=ODHR6AX3XXXUZ3SQH5STHOEVLOXDLCEEXER6CAZ2GTFH5ZMIG3GQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160103556 (541h58m3.031186591s) [noise=true , address=145.79.227.11:4730 , piece_id=YQ3EPI5WFTRKZYMBETNB4Z5OFJH2EYO4D3K5IUPPUK5LOPG5HR7A, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160540639 (541h58m3.031623684s) [noise=true , address=77.90.31.13:4626 , piece_id=SWWTTWZXFDJYVPGXC2BKP7F2HBMEWAQH3K3CWRNFXJ3TSYIJ27SA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160647302 (541h58m3.031730347s) [noise=true , address=145.79.227.48:4301 , piece_id=F4S27FRU5QEW6773ETE5LUHGJURUI7Q6LL37LQ5A7ZJ5FGJRNGBQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160963424 (541h58m3.032046489s) [noise=true , address=145.79.227.33:4869 , piece_id=MMV6BYYZXSVJMCPED5JSGVTF4JSTR3QHXNUW5AMTB2ZE3UZHJKUA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161318821 (541h58m3.032401866s) [noise=true , address=145.79.227.18:4006 , piece_id=CFEWKLHYZCVFA5QMLUHBOOPDYJI5QBXY62YOE3CDTDMWDJLTC2BA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161411347 (541h58m3.032494392s) [noise=true , address=145.79.227.52:4447 , piece_id=2O3HLQNMHZ2O3PLBJ3DWOIANEYV67KVSGH54WC2NNA7W3RPHITDA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161744611 (541h58m3.032827656s) [noise=true , address=145.79.227.10:4401 , piece_id=YEFNE6O7V5AUE4IPM3I37PISHSRDS7V4M76SNWSCGNHISZGJDL6Q, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161982905 (541h58m3.03306595s) [noise=true , address=145.79.227.17:4971 , piece_id=3764RIOYIQK55MXDBWUF546CDCZURC5OYVEKJPPTHLYU6AGWI5LQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.162231348 (541h58m3.033314393s) [noise=true , address=77.90.31.18:4805 , piece_id=NIHNZJSLKSOHKQ4PVXGRQV6X5FPMG5ZEBEVSB7YZODSR77FG4USQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.162713767 (541h58m3.033796802s) [noise=true , address=145.79.227.2:4115 , piece_id=7Q3ZBUPUZZKCYTMQXPMMOAY72M22SWDEIK6ZMGQX4X7AJL2VSTVA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.163411746 (541h58m3.034494791s) [noise=true , address=145.79.227.47:4267 , piece_id=OAECK63J4JA4PSR27KLMVJGQ2RJJTJC2ZCZK7DADWNF2LGPGSN4A, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.164102211 (541h58m3.035185266s) [noise=true , address=145.79.227.31:4798 , piece_id=FCBBBUGAZSKCUMQUBEQH5YMEZYXHDZL72DQNX3FYURLNDYL2BC6A, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:04.792807454 (541h58m4.663890499s) [upload=475268]: Next returned nil segment Dec 28 05:27:04.792913676 (541h58m4.663996731s) [upload=475268]: Waiting for error group managing segments... Dec 28 05:27:05.170090612 (541h58m5.041173647s) [noise=true , address=145.79.227.18:4006 , piece_id=CFEWKLHYZCVFA5QMLUHBOOPDYJI5QBXY62YOE3CDTDMWDJLTC2BA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.174952089 (541h58m5.046035134s) [noise=true , address=145.79.227.17:4971 , piece_id=3764RIOYIQK55MXDBWUF546CDCZURC5OYVEKJPPTHLYU6AGWI5LQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.177591725 (541h58m5.04867477s) [noise=true , address=145.79.227.31:4798 , piece_id=FCBBBUGAZSKCUMQUBEQH5YMEZYXHDZL72DQNX3FYURLNDYL2BC6A, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.178981341 (541h58m5.050064386s) [noise=true , address=145.79.227.10:4401 , piece_id=YEFNE6O7V5AUE4IPM3I37PISHSRDS7V4M76SNWSCGNHISZGJDL6Q, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.179298094 (541h58m5.050381129s) [noise=true , address=145.79.227.11:4730 , piece_id=YQ3EPI5WFTRKZYMBETNB4Z5OFJH2EYO4D3K5IUPPUK5LOPG5HR7A, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.180046579 (541h58m5.051129624s) [noise=true , address=145.79.227.50:4373 , piece_id=7QUOTLRQTUHNDIQXVUHOHBPJI3XM5MOCQN2KQZV4N3ZRLCRWOZZA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.181848801 (541h58m5.052931846s) [noise=true , address=145.79.227.48:4301 , piece_id=F4S27FRU5QEW6773ETE5LUHGJURUI7Q6LL37LQ5A7ZJ5FGJRNGBQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.184752010 (541h58m5.055835055s) [noise=true , address=145.79.227.52:4447 , piece_id=2O3HLQNMHZ2O3PLBJ3DWOIANEYV67KVSGH54WC2NNA7W3RPHITDA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.187406204 (541h58m5.058489309s) [noise=true , address=145.79.227.47:4267 , piece_id=OAECK63J4JA4PSR27KLMVJGQ2RJJTJC2ZCZK7DADWNF2LGPGSN4A, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.187473783 (541h58m5.058556818s) [noise=true , address=145.79.227.15:4873 , piece_id=ODHR6AX3XXXUZ3SQH5STHOEVLOXDLCEEXER6CAZ2GTFH5ZMIG3GQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.200556825 (541h58m5.07163987s) [noise=true , address=145.79.227.2:4115 , piece_id=7Q3ZBUPUZZKCYTMQXPMMOAY72M22SWDEIK6ZMGQX4X7AJL2VSTVA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.452923030 (541h58m5.324006065s) [noise=true , address=77.90.31.18:4805 , piece_id=NIHNZJSLKSOHKQ4PVXGRQV6X5FPMG5ZEBEVSB7YZODSR77FG4USQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.462509354 (541h58m5.333592399s) [noise=true , address=77.90.31.13:4626 , piece_id=SWWTTWZXFDJYVPGXC2BKP7F2HBMEWAQH3K3CWRNFXJ3TSYIJ27SA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.462707562 (541h58m5.333790607s) [seg_pos={0 0}, upload=475268]: Segment reached optimal threshold of 13 pieces. Dec 28 05:27:05.462920056 (541h58m5.334003101s) [noise=true , address=145.79.227.44:4195 , piece_id=TK6S7EW4Q63JFRRLE6UPPREIFZNJNNIFIO4KR2WDLIXNSD6USSSA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 05:27:05.462929304 (541h58m5.33401235s) [noise=true , address=145.79.227.44:4195 , piece_id=TK6S7EW4Q63JFRRLE6UPPREIFZNJNNIFIO4KR2WDLIXNSD6USSSA, seg_pos={0 0}, upload=475268]: Long tail cancellation due to optimal threshold Dec 28 05:27:05.462933182 (541h58m5.334016247s) [noise=true , address=145.79.227.33:4869 , piece_id=MMV6BYYZXSVJMCPED5JSGVTF4JSTR3QHXNUW5AMTB2ZE3UZHJKUA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 05:27:05.462943321 (541h58m5.334026376s) [noise=true , address=145.79.227.33:4869 , piece_id=MMV6BYYZXSVJMCPED5JSGVTF4JSTR3QHXNUW5AMTB2ZE3UZHJKUA, seg_pos={0 0}, upload=475268]: Long tail cancellation due to optimal threshold Dec 28 05:27:05.463111251 (541h58m5.334194286s) [noise=true , address=45.91.116.8:4048 , piece_id=XFGIU5DBLMG3CQBAKWUX5SP4RKDSBPLCDBE6I2X525IGLM5I6LWQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 05:27:05.463117372 (541h58m5.334200417s) [noise=true , address=45.91.116.8:4048 , piece_id=XFGIU5DBLMG3CQBAKWUX5SP4RKDSBPLCDBE6I2X525IGLM5I6LWQ, seg_pos={0 0}, upload=475268]: Long tail cancellation due to optimal threshold Dec 28 05:27:05.463138072 (541h58m5.334221107s) [seg_pos={0 0}, upload=475268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:27:05.463241077 (541h58m5.334324112s) [upload=475268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:27:05.463246508 (541h58m5.334329543s) [upload=475268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:27:05.762183782 (541h58m5.633266817s) [upload=475268]: Done uploading segments... Dec 28 05:27:05.762209060 (541h58m5.633292105s) [upload=475268]: Upload finished. err: <nil> Duration: 6.09320677s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.002668121s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:27:39 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.002698349s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.X] duration: 30.003474461s trace: https://jaeger.dev.storj.io/trace/3b186855697d524c FAIL (2025-12-28 05:27:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:27:02.906913362 (541h58m2.777996397s) [upload=475268]: Starting upload Dec 28 05:27:02.906987653 (541h58m2.778070698s) [upload=475268]: Done starting upload Dec 28 05:27:02.907112942 (541h58m2.778195997s) [upload=475268]: Uploading segments... Dec 28 05:27:02.907210587 (541h58m2.778293632s) [upload=475268]: Got next segment. Inline: false Dec 28 05:27:02.907331678 (541h58m2.778414723s) [upload=475268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:27:02.907338391 (541h58m2.778421436s) [upload=475268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:27:03.158542814 (541h58m3.029625879s) [seg_pos={0 0}, upload=475268]: Begin upload segment... Dec 28 05:27:03.158766371 (541h58m3.029849416s) [seg_pos={0 0}, upload=475268]: Done begin upload segment. Dec 28 05:27:03.158809653 (541h58m3.029892698s) [noise=true , address=45.91.116.8:4048 , piece_id=XFGIU5DBLMG3CQBAKWUX5SP4RKDSBPLCDBE6I2X525IGLM5I6LWQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.159261533 (541h58m3.030344578s) [noise=true , address=145.79.227.50:4373 , piece_id=7QUOTLRQTUHNDIQXVUHOHBPJI3XM5MOCQN2KQZV4N3ZRLCRWOZZA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.159652347 (541h58m3.030735392s) [noise=true , address=145.79.227.44:4195 , piece_id=TK6S7EW4Q63JFRRLE6UPPREIFZNJNNIFIO4KR2WDLIXNSD6USSSA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.159843091 (541h58m3.030926136s) [noise=true , address=145.79.227.15:4873 , piece_id=ODHR6AX3XXXUZ3SQH5STHOEVLOXDLCEEXER6CAZ2GTFH5ZMIG3GQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160103556 (541h58m3.031186591s) [noise=true , address=145.79.227.11:4730 , piece_id=YQ3EPI5WFTRKZYMBETNB4Z5OFJH2EYO4D3K5IUPPUK5LOPG5HR7A, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160540639 (541h58m3.031623684s) [noise=true , address=77.90.31.13:4626 , piece_id=SWWTTWZXFDJYVPGXC2BKP7F2HBMEWAQH3K3CWRNFXJ3TSYIJ27SA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160647302 (541h58m3.031730347s) [noise=true , address=145.79.227.48:4301 , piece_id=F4S27FRU5QEW6773ETE5LUHGJURUI7Q6LL37LQ5A7ZJ5FGJRNGBQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.160963424 (541h58m3.032046489s) [noise=true , address=145.79.227.33:4869 , piece_id=MMV6BYYZXSVJMCPED5JSGVTF4JSTR3QHXNUW5AMTB2ZE3UZHJKUA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161318821 (541h58m3.032401866s) [noise=true , address=145.79.227.18:4006 , piece_id=CFEWKLHYZCVFA5QMLUHBOOPDYJI5QBXY62YOE3CDTDMWDJLTC2BA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161411347 (541h58m3.032494392s) [noise=true , address=145.79.227.52:4447 , piece_id=2O3HLQNMHZ2O3PLBJ3DWOIANEYV67KVSGH54WC2NNA7W3RPHITDA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161744611 (541h58m3.032827656s) [noise=true , address=145.79.227.10:4401 , piece_id=YEFNE6O7V5AUE4IPM3I37PISHSRDS7V4M76SNWSCGNHISZGJDL6Q, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.161982905 (541h58m3.03306595s) [noise=true , address=145.79.227.17:4971 , piece_id=3764RIOYIQK55MXDBWUF546CDCZURC5OYVEKJPPTHLYU6AGWI5LQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.162231348 (541h58m3.033314393s) [noise=true , address=77.90.31.18:4805 , piece_id=NIHNZJSLKSOHKQ4PVXGRQV6X5FPMG5ZEBEVSB7YZODSR77FG4USQ, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.162713767 (541h58m3.033796802s) [noise=true , address=145.79.227.2:4115 , piece_id=7Q3ZBUPUZZKCYTMQXPMMOAY72M22SWDEIK6ZMGQX4X7AJL2VSTVA, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.163411746 (541h58m3.034494791s) [noise=true , address=145.79.227.47:4267 , piece_id=OAECK63J4JA4PSR27KLMVJGQ2RJJTJC2ZCZK7DADWNF2LGPGSN4A, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:03.164102211 (541h58m3.035185266s) [noise=true , address=145.79.227.31:4798 , piece_id=FCBBBUGAZSKCUMQUBEQH5YMEZYXHDZL72DQNX3FYURLNDYL2BC6A, seg_pos={0 0}, upload=475268]: Uploading piece... Dec 28 05:27:04.792807454 (541h58m4.663890499s) [upload=475268]: Next returned nil segment Dec 28 05:27:04.792913676 (541h58m4.663996731s) [upload=475268]: Waiting for error group managing segments... Dec 28 05:27:05.170090612 (541h58m5.041173647s) [noise=true , address=145.79.227.18:4006 , piece_id=CFEWKLHYZCVFA5QMLUHBOOPDYJI5QBXY62YOE3CDTDMWDJLTC2BA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.174952089 (541h58m5.046035134s) [noise=true , address=145.79.227.17:4971 , piece_id=3764RIOYIQK55MXDBWUF546CDCZURC5OYVEKJPPTHLYU6AGWI5LQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.177591725 (541h58m5.04867477s) [noise=true , address=145.79.227.31:4798 , piece_id=FCBBBUGAZSKCUMQUBEQH5YMEZYXHDZL72DQNX3FYURLNDYL2BC6A, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.178981341 (541h58m5.050064386s) [noise=true , address=145.79.227.10:4401 , piece_id=YEFNE6O7V5AUE4IPM3I37PISHSRDS7V4M76SNWSCGNHISZGJDL6Q, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.179298094 (541h58m5.050381129s) [noise=true , address=145.79.227.11:4730 , piece_id=YQ3EPI5WFTRKZYMBETNB4Z5OFJH2EYO4D3K5IUPPUK5LOPG5HR7A, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.180046579 (541h58m5.051129624s) [noise=true , address=145.79.227.50:4373 , piece_id=7QUOTLRQTUHNDIQXVUHOHBPJI3XM5MOCQN2KQZV4N3ZRLCRWOZZA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.181848801 (541h58m5.052931846s) [noise=true , address=145.79.227.48:4301 , piece_id=F4S27FRU5QEW6773ETE5LUHGJURUI7Q6LL37LQ5A7ZJ5FGJRNGBQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.184752010 (541h58m5.055835055s) [noise=true , address=145.79.227.52:4447 , piece_id=2O3HLQNMHZ2O3PLBJ3DWOIANEYV67KVSGH54WC2NNA7W3RPHITDA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.187406204 (541h58m5.058489309s) [noise=true , address=145.79.227.47:4267 , piece_id=OAECK63J4JA4PSR27KLMVJGQ2RJJTJC2ZCZK7DADWNF2LGPGSN4A, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.187473783 (541h58m5.058556818s) [noise=true , address=145.79.227.15:4873 , piece_id=ODHR6AX3XXXUZ3SQH5STHOEVLOXDLCEEXER6CAZ2GTFH5ZMIG3GQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.200556825 (541h58m5.07163987s) [noise=true , address=145.79.227.2:4115 , piece_id=7Q3ZBUPUZZKCYTMQXPMMOAY72M22SWDEIK6ZMGQX4X7AJL2VSTVA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.452923030 (541h58m5.324006065s) [noise=true , address=77.90.31.18:4805 , piece_id=NIHNZJSLKSOHKQ4PVXGRQV6X5FPMG5ZEBEVSB7YZODSR77FG4USQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.462509354 (541h58m5.333592399s) [noise=true , address=77.90.31.13:4626 , piece_id=SWWTTWZXFDJYVPGXC2BKP7F2HBMEWAQH3K3CWRNFXJ3TSYIJ27SA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: <nil> Dec 28 05:27:05.462707562 (541h58m5.333790607s) [seg_pos={0 0}, upload=475268]: Segment reached optimal threshold of 13 pieces. Dec 28 05:27:05.462920056 (541h58m5.334003101s) [noise=true , address=145.79.227.44:4195 , piece_id=TK6S7EW4Q63JFRRLE6UPPREIFZNJNNIFIO4KR2WDLIXNSD6USSSA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 05:27:05.462929304 (541h58m5.33401235s) [noise=true , address=145.79.227.44:4195 , piece_id=TK6S7EW4Q63JFRRLE6UPPREIFZNJNNIFIO4KR2WDLIXNSD6USSSA, seg_pos={0 0}, upload=475268]: Long tail cancellation due to optimal threshold Dec 28 05:27:05.462933182 (541h58m5.334016247s) [noise=true , address=145.79.227.33:4869 , piece_id=MMV6BYYZXSVJMCPED5JSGVTF4JSTR3QHXNUW5AMTB2ZE3UZHJKUA, seg_pos={0 0}, upload=475268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 05:27:05.462943321 (541h58m5.334026376s) [noise=true , address=145.79.227.33:4869 , piece_id=MMV6BYYZXSVJMCPED5JSGVTF4JSTR3QHXNUW5AMTB2ZE3UZHJKUA, seg_pos={0 0}, upload=475268]: Long tail cancellation due to optimal threshold Dec 28 05:27:05.463111251 (541h58m5.334194286s) [noise=true , address=45.91.116.8:4048 , piece_id=XFGIU5DBLMG3CQBAKWUX5SP4RKDSBPLCDBE6I2X525IGLM5I6LWQ, seg_pos={0 0}, upload=475268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 05:27:05.463117372 (541h58m5.334200417s) [noise=true , address=45.91.116.8:4048 , piece_id=XFGIU5DBLMG3CQBAKWUX5SP4RKDSBPLCDBE6I2X525IGLM5I6LWQ, seg_pos={0 0}, upload=475268]: Long tail cancellation due to optimal threshold Dec 28 05:27:05.463138072 (541h58m5.334221107s) [seg_pos={0 0}, upload=475268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:27:05.463241077 (541h58m5.334324112s) [upload=475268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:27:05.463246508 (541h58m5.334329543s) [upload=475268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:27:05.762183782 (541h58m5.633266817s) [upload=475268]: Done uploading segments... Dec 28 05:27:05.762209060 (541h58m5.633292105s) [upload=475268]: Upload finished. err: <nil> Duration: 30.003498546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....X] duration: 5.001703689s trace: https://jaeger.dev.storj.io/trace/3fae4afb97c7f392 FAIL (2025-12-28 05:27:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001780436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............................X.......] duration: 41.22995393s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 41.229981241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.238176355s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:26:48 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.23820503s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X.] duration: 5.650417259s trace: https://jaeger.dev.storj.io/trace/557e9c5abd8d7599 FAIL (2025-12-28 05:26:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:24:54.331996620 (541h55m54.203079666s) [upload=475239]: Starting upload Dec 28 05:24:54.332076312 (541h55m54.203159357s) [upload=475239]: Done starting upload Dec 28 05:24:54.332225566 (541h55m54.203308601s) [upload=475239]: Uploading segments... Dec 28 05:24:54.332335545 (541h55m54.20341858s) [upload=475239]: Got next segment. Inline: false Dec 28 05:24:54.332369279 (541h55m54.203452314s) [upload=475239]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:24:54.332373597 (541h55m54.203456633s) [upload=475239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:24:54.577229252 (541h55m54.448312287s) [seg_pos={0 0}, upload=475239]: Begin upload segment... Dec 28 05:24:54.577308252 (541h55m54.448391287s) [seg_pos={0 0}, upload=475239]: Done begin upload segment. Dec 28 05:24:54.577385359 (541h55m54.448468404s) [noise=true , address=145.79.227.32:4836 , piece_id=6SWULBI2ZHD5IVM7BORO4WOOHIISFDL7AJSQ4QVTUWNQS5L2ES7A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.577454470 (541h55m54.448537515s) [noise=true , address=145.79.227.47:4266 , piece_id=4AITQPPE6NGGKAKU5XCHOAODDNJTQQ6RWFOEYLSWVNJ3T6ZESC4A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.578078509 (541h55m54.449161554s) [noise=true , address=145.79.227.24:4583 , piece_id=QV5FEVGM4JOJ7CBGGPTLQR75O2ULZEJ74AD7EFPOOLK5XXQGSFEQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.578369863 (541h55m54.449452908s) [noise=true , address=145.79.227.36:4518 , piece_id=OE2RIMNRQ73CXVT6UDKAVGG2FA6FLRQYVTDR3DRYC7XICFDCMOZA, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.578990195 (541h55m54.45007323s) [noise=true , address=145.79.227.48:4301 , piece_id=KS27XH2INP5ESSLL7I2VZIO7US2SGX75D6U4P2HUQUV5QQJRVMWQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579040751 (541h55m54.450123786s) [noise=true , address=145.79.227.37:4567 , piece_id=FA6BWGTPUZBUG3OTH3WZ6O5ZJONNIY2KD4INXXZH2O7TGZGY4BXQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579394224 (541h55m54.450477269s) [noise=true , address=145.79.227.5:4222 , piece_id=SMT63GQAQRXRCUZ5LF2UOLA4BBMMXLCO7NNHXSL5DDAOJBI5LYCQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579494916 (541h55m54.450577961s) [noise=true , address=145.79.227.16:4911 , piece_id=KXVG4XVUWRE6ZNQAP2UK3ET6E64SACRDHUFNB5HH6OJQG2F4H5LA, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579782774 (541h55m54.450865819s) [noise=true , address=45.91.116.5:4297 , piece_id=U5A7E3TFCGDIRKZ5AOKCZGU2YSYYR7FH2K6BNNHJPPGJJTAR2VDQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579857346 (541h55m54.450940381s) [noise=true , address=45.91.116.11:4117 , piece_id=VW4UEV5VGMLZCGKW6FM7F5XENZ6AKXAJ6URZFY7YDWBTY7APVHPQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580184940 (541h55m54.451267975s) [noise=true , address=45.91.116.10:4081 , piece_id=PVYN5DT2WVYKPU6SI35GY2Q7YO4IMXB73JVOKRDIY6QJ7BDYKLGQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580419656 (541h55m54.451502701s) [noise=true , address=145.79.227.19:4077 , piece_id=S2MLIAXPPVPPAJSDCYUWE6NZ3NXVM6QBCRGGPG4QTI562GZ43JUQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580612264 (541h55m54.451695309s) [noise=true , address=77.90.31.16:4735 , piece_id=JZ5TYGFA7KDURBFQRUE4FTDTAHPKMFVOMEFYSGNV2IBU22KKTI2A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580785564 (541h55m54.451868609s) [noise=true , address=45.91.116.12:4228 , piece_id=4XEPMEYWH62GXLCJF4OVSPKJ4V6G5WTRK2CJFLGCUF3B2762UF7A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.581046039 (541h55m54.452129084s) [noise=true , address=77.90.31.21:4878 , piece_id=O5HZRJD4VURRQT26HFMBM3JRMOXVLG6DHEL553GTDURF4QIQKY3Q, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.581168393 (541h55m54.452251428s) [noise=true , address=145.79.227.11:4731 , piece_id=3HOJMVGCHN47WFU5ZYLXJC4OTNODMWHGF4DE5Y26FCKSSSLVAQTA, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:56.311493668 (541h55m56.182576703s) [upload=475239]: Next returned nil segment Dec 28 05:24:56.311508527 (541h55m56.182591562s) [upload=475239]: Waiting for error group managing segments... Dec 28 05:24:56.589094284 (541h55m56.460177319s) [noise=true , address=145.79.227.36:4518 , piece_id=OE2RIMNRQ73CXVT6UDKAVGG2FA6FLRQYVTDR3DRYC7XICFDCMOZA, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.597941440 (541h55m56.469024475s) [noise=true , address=145.79.227.16:4911 , piece_id=KXVG4XVUWRE6ZNQAP2UK3ET6E64SACRDHUFNB5HH6OJQG2F4H5LA, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.600636884 (541h55m56.471719919s) [noise=true , address=145.79.227.24:4583 , piece_id=QV5FEVGM4JOJ7CBGGPTLQR75O2ULZEJ74AD7EFPOOLK5XXQGSFEQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.600946252 (541h55m56.472029287s) [noise=true , address=145.79.227.47:4266 , piece_id=4AITQPPE6NGGKAKU5XCHOAODDNJTQQ6RWFOEYLSWVNJ3T6ZESC4A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.623334413 (541h55m56.494417458s) [noise=true , address=145.79.227.11:4731 , piece_id=3HOJMVGCHN47WFU5ZYLXJC4OTNODMWHGF4DE5Y26FCKSSSLVAQTA, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.637583956 (541h55m56.508667001s) [noise=true , address=145.79.227.32:4836 , piece_id=6SWULBI2ZHD5IVM7BORO4WOOHIISFDL7AJSQ4QVTUWNQS5L2ES7A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.640032018 (541h55m56.511115063s) [noise=true , address=145.79.227.37:4567 , piece_id=FA6BWGTPUZBUG3OTH3WZ6O5ZJONNIY2KD4INXXZH2O7TGZGY4BXQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.641628037 (541h55m56.512711072s) [noise=true , address=145.79.227.48:4301 , piece_id=KS27XH2INP5ESSLL7I2VZIO7US2SGX75D6U4P2HUQUV5QQJRVMWQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.643139856 (541h55m56.514222931s) [noise=true , address=145.79.227.5:4222 , piece_id=SMT63GQAQRXRCUZ5LF2UOLA4BBMMXLCO7NNHXSL5DDAOJBI5LYCQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.774156472 (541h55m56.645239507s) [noise=true , address=77.90.31.21:4878 , piece_id=O5HZRJD4VURRQT26HFMBM3JRMOXVLG6DHEL553GTDURF4QIQKY3Q, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.804098496 (541h55m56.675181541s) [noise=true , address=145.79.227.19:4077 , piece_id=S2MLIAXPPVPPAJSDCYUWE6NZ3NXVM6QBCRGGPG4QTI562GZ43JUQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.822381500 (541h55m56.693464545s) [noise=true , address=77.90.31.16:4735 , piece_id=JZ5TYGFA7KDURBFQRUE4FTDTAHPKMFVOMEFYSGNV2IBU22KKTI2A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.830994441 (541h55m56.702077486s) [noise=true , address=45.91.116.12:4228 , piece_id=4XEPMEYWH62GXLCJF4OVSPKJ4V6G5WTRK2CJFLGCUF3B2762UF7A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.831019898 (541h55m56.702102933s) [seg_pos={0 0}, upload=475239]: Segment reached optimal threshold of 13 pieces. Dec 28 05:24:56.831248644 (541h55m56.702331679s) [noise=true , address=45.91.116.11:4117 , piece_id=VW4UEV5VGMLZCGKW6FM7F5XENZ6AKXAJ6URZFY7YDWBTY7APVHPQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 05:24:56.831256730 (541h55m56.702339775s) [noise=true , address=45.91.116.11:4117 , piece_id=VW4UEV5VGMLZCGKW6FM7F5XENZ6AKXAJ6URZFY7YDWBTY7APVHPQ, seg_pos={0 0}, upload=475239]: Long tail cancellation due to optimal threshold Dec 28 05:24:56.831289151 (541h55m56.702372186s) [noise=true , address=45.91.116.5:4297 , piece_id=U5A7E3TFCGDIRKZ5AOKCZGU2YSYYR7FH2K6BNNHJPPGJJTAR2VDQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 05:24:56.831298479 (541h55m56.702381514s) [noise=true , address=45.91.116.5:4297 , piece_id=U5A7E3TFCGDIRKZ5AOKCZGU2YSYYR7FH2K6BNNHJPPGJJTAR2VDQ, seg_pos={0 0}, upload=475239]: Long tail cancellation due to optimal threshold Dec 28 05:24:56.831447042 (541h55m56.702530078s) [noise=true , address=45.91.116.10:4081 , piece_id=PVYN5DT2WVYKPU6SI35GY2Q7YO4IMXB73JVOKRDIY6QJ7BDYKLGQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 05:24:56.831455268 (541h55m56.702538313s) [noise=true , address=45.91.116.10:4081 , piece_id=PVYN5DT2WVYKPU6SI35GY2Q7YO4IMXB73JVOKRDIY6QJ7BDYKLGQ, seg_pos={0 0}, upload=475239]: Long tail cancellation due to optimal threshold Dec 28 05:24:56.831479524 (541h55m56.702562569s) [seg_pos={0 0}, upload=475239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:24:56.831556170 (541h55m56.702639215s) [upload=475239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:24:56.831561009 (541h55m56.702644044s) [upload=475239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:24:57.124127042 (541h55m56.995210087s) [upload=475239]: Done uploading segments... Dec 28 05:24:57.124149815 (541h55m56.99523286s) [upload=475239]: Upload finished. err: <nil> Duration: 5.65044809s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.181907032s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:26:08 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.181933873s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.............................X......] duration: 48.565707533s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 48.565723783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX....] duration: 3.524061809s trace: https://jaeger.dev.storj.io/trace/2d0c9871e86208dd FAIL (2025-12-28 05:25:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524091044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.966431219s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:25:25 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.966462949s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...X] duration: 30.002992018s trace: https://jaeger.dev.storj.io/trace/557e9c5abd8d7599 FAIL (2025-12-28 05:25:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:24:54.331996620 (541h55m54.203079666s) [upload=475239]: Starting upload Dec 28 05:24:54.332076312 (541h55m54.203159357s) [upload=475239]: Done starting upload Dec 28 05:24:54.332225566 (541h55m54.203308601s) [upload=475239]: Uploading segments... Dec 28 05:24:54.332335545 (541h55m54.20341858s) [upload=475239]: Got next segment. Inline: false Dec 28 05:24:54.332369279 (541h55m54.203452314s) [upload=475239]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:24:54.332373597 (541h55m54.203456633s) [upload=475239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:24:54.577229252 (541h55m54.448312287s) [seg_pos={0 0}, upload=475239]: Begin upload segment... Dec 28 05:24:54.577308252 (541h55m54.448391287s) [seg_pos={0 0}, upload=475239]: Done begin upload segment. Dec 28 05:24:54.577385359 (541h55m54.448468404s) [noise=true , address=145.79.227.32:4836 , piece_id=6SWULBI2ZHD5IVM7BORO4WOOHIISFDL7AJSQ4QVTUWNQS5L2ES7A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.577454470 (541h55m54.448537515s) [noise=true , address=145.79.227.47:4266 , piece_id=4AITQPPE6NGGKAKU5XCHOAODDNJTQQ6RWFOEYLSWVNJ3T6ZESC4A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.578078509 (541h55m54.449161554s) [noise=true , address=145.79.227.24:4583 , piece_id=QV5FEVGM4JOJ7CBGGPTLQR75O2ULZEJ74AD7EFPOOLK5XXQGSFEQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.578369863 (541h55m54.449452908s) [noise=true , address=145.79.227.36:4518 , piece_id=OE2RIMNRQ73CXVT6UDKAVGG2FA6FLRQYVTDR3DRYC7XICFDCMOZA, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.578990195 (541h55m54.45007323s) [noise=true , address=145.79.227.48:4301 , piece_id=KS27XH2INP5ESSLL7I2VZIO7US2SGX75D6U4P2HUQUV5QQJRVMWQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579040751 (541h55m54.450123786s) [noise=true , address=145.79.227.37:4567 , piece_id=FA6BWGTPUZBUG3OTH3WZ6O5ZJONNIY2KD4INXXZH2O7TGZGY4BXQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579394224 (541h55m54.450477269s) [noise=true , address=145.79.227.5:4222 , piece_id=SMT63GQAQRXRCUZ5LF2UOLA4BBMMXLCO7NNHXSL5DDAOJBI5LYCQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579494916 (541h55m54.450577961s) [noise=true , address=145.79.227.16:4911 , piece_id=KXVG4XVUWRE6ZNQAP2UK3ET6E64SACRDHUFNB5HH6OJQG2F4H5LA, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579782774 (541h55m54.450865819s) [noise=true , address=45.91.116.5:4297 , piece_id=U5A7E3TFCGDIRKZ5AOKCZGU2YSYYR7FH2K6BNNHJPPGJJTAR2VDQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.579857346 (541h55m54.450940381s) [noise=true , address=45.91.116.11:4117 , piece_id=VW4UEV5VGMLZCGKW6FM7F5XENZ6AKXAJ6URZFY7YDWBTY7APVHPQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580184940 (541h55m54.451267975s) [noise=true , address=45.91.116.10:4081 , piece_id=PVYN5DT2WVYKPU6SI35GY2Q7YO4IMXB73JVOKRDIY6QJ7BDYKLGQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580419656 (541h55m54.451502701s) [noise=true , address=145.79.227.19:4077 , piece_id=S2MLIAXPPVPPAJSDCYUWE6NZ3NXVM6QBCRGGPG4QTI562GZ43JUQ, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580612264 (541h55m54.451695309s) [noise=true , address=77.90.31.16:4735 , piece_id=JZ5TYGFA7KDURBFQRUE4FTDTAHPKMFVOMEFYSGNV2IBU22KKTI2A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.580785564 (541h55m54.451868609s) [noise=true , address=45.91.116.12:4228 , piece_id=4XEPMEYWH62GXLCJF4OVSPKJ4V6G5WTRK2CJFLGCUF3B2762UF7A, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.581046039 (541h55m54.452129084s) [noise=true , address=77.90.31.21:4878 , piece_id=O5HZRJD4VURRQT26HFMBM3JRMOXVLG6DHEL553GTDURF4QIQKY3Q, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:54.581168393 (541h55m54.452251428s) [noise=true , address=145.79.227.11:4731 , piece_id=3HOJMVGCHN47WFU5ZYLXJC4OTNODMWHGF4DE5Y26FCKSSSLVAQTA, seg_pos={0 0}, upload=475239]: Uploading piece... Dec 28 05:24:56.311493668 (541h55m56.182576703s) [upload=475239]: Next returned nil segment Dec 28 05:24:56.311508527 (541h55m56.182591562s) [upload=475239]: Waiting for error group managing segments... Dec 28 05:24:56.589094284 (541h55m56.460177319s) [noise=true , address=145.79.227.36:4518 , piece_id=OE2RIMNRQ73CXVT6UDKAVGG2FA6FLRQYVTDR3DRYC7XICFDCMOZA, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.597941440 (541h55m56.469024475s) [noise=true , address=145.79.227.16:4911 , piece_id=KXVG4XVUWRE6ZNQAP2UK3ET6E64SACRDHUFNB5HH6OJQG2F4H5LA, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.600636884 (541h55m56.471719919s) [noise=true , address=145.79.227.24:4583 , piece_id=QV5FEVGM4JOJ7CBGGPTLQR75O2ULZEJ74AD7EFPOOLK5XXQGSFEQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.600946252 (541h55m56.472029287s) [noise=true , address=145.79.227.47:4266 , piece_id=4AITQPPE6NGGKAKU5XCHOAODDNJTQQ6RWFOEYLSWVNJ3T6ZESC4A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.623334413 (541h55m56.494417458s) [noise=true , address=145.79.227.11:4731 , piece_id=3HOJMVGCHN47WFU5ZYLXJC4OTNODMWHGF4DE5Y26FCKSSSLVAQTA, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.637583956 (541h55m56.508667001s) [noise=true , address=145.79.227.32:4836 , piece_id=6SWULBI2ZHD5IVM7BORO4WOOHIISFDL7AJSQ4QVTUWNQS5L2ES7A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.640032018 (541h55m56.511115063s) [noise=true , address=145.79.227.37:4567 , piece_id=FA6BWGTPUZBUG3OTH3WZ6O5ZJONNIY2KD4INXXZH2O7TGZGY4BXQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.641628037 (541h55m56.512711072s) [noise=true , address=145.79.227.48:4301 , piece_id=KS27XH2INP5ESSLL7I2VZIO7US2SGX75D6U4P2HUQUV5QQJRVMWQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.643139856 (541h55m56.514222931s) [noise=true , address=145.79.227.5:4222 , piece_id=SMT63GQAQRXRCUZ5LF2UOLA4BBMMXLCO7NNHXSL5DDAOJBI5LYCQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.774156472 (541h55m56.645239507s) [noise=true , address=77.90.31.21:4878 , piece_id=O5HZRJD4VURRQT26HFMBM3JRMOXVLG6DHEL553GTDURF4QIQKY3Q, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.804098496 (541h55m56.675181541s) [noise=true , address=145.79.227.19:4077 , piece_id=S2MLIAXPPVPPAJSDCYUWE6NZ3NXVM6QBCRGGPG4QTI562GZ43JUQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.822381500 (541h55m56.693464545s) [noise=true , address=77.90.31.16:4735 , piece_id=JZ5TYGFA7KDURBFQRUE4FTDTAHPKMFVOMEFYSGNV2IBU22KKTI2A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.830994441 (541h55m56.702077486s) [noise=true , address=45.91.116.12:4228 , piece_id=4XEPMEYWH62GXLCJF4OVSPKJ4V6G5WTRK2CJFLGCUF3B2762UF7A, seg_pos={0 0}, upload=475239]: Done uploading piece. err: <nil> Dec 28 05:24:56.831019898 (541h55m56.702102933s) [seg_pos={0 0}, upload=475239]: Segment reached optimal threshold of 13 pieces. Dec 28 05:24:56.831248644 (541h55m56.702331679s) [noise=true , address=45.91.116.11:4117 , piece_id=VW4UEV5VGMLZCGKW6FM7F5XENZ6AKXAJ6URZFY7YDWBTY7APVHPQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 05:24:56.831256730 (541h55m56.702339775s) [noise=true , address=45.91.116.11:4117 , piece_id=VW4UEV5VGMLZCGKW6FM7F5XENZ6AKXAJ6URZFY7YDWBTY7APVHPQ, seg_pos={0 0}, upload=475239]: Long tail cancellation due to optimal threshold Dec 28 05:24:56.831289151 (541h55m56.702372186s) [noise=true , address=45.91.116.5:4297 , piece_id=U5A7E3TFCGDIRKZ5AOKCZGU2YSYYR7FH2K6BNNHJPPGJJTAR2VDQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 05:24:56.831298479 (541h55m56.702381514s) [noise=true , address=45.91.116.5:4297 , piece_id=U5A7E3TFCGDIRKZ5AOKCZGU2YSYYR7FH2K6BNNHJPPGJJTAR2VDQ, seg_pos={0 0}, upload=475239]: Long tail cancellation due to optimal threshold Dec 28 05:24:56.831447042 (541h55m56.702530078s) [noise=true , address=45.91.116.10:4081 , piece_id=PVYN5DT2WVYKPU6SI35GY2Q7YO4IMXB73JVOKRDIY6QJ7BDYKLGQ, seg_pos={0 0}, upload=475239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 05:24:56.831455268 (541h55m56.702538313s) [noise=true , address=45.91.116.10:4081 , piece_id=PVYN5DT2WVYKPU6SI35GY2Q7YO4IMXB73JVOKRDIY6QJ7BDYKLGQ, seg_pos={0 0}, upload=475239]: Long tail cancellation due to optimal threshold Dec 28 05:24:56.831479524 (541h55m56.702562569s) [seg_pos={0 0}, upload=475239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:24:56.831556170 (541h55m56.702639215s) [upload=475239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:24:56.831561009 (541h55m56.702644044s) [upload=475239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:24:57.124127042 (541h55m56.995210087s) [upload=475239]: Done uploading segments... Dec 28 05:24:57.124149815 (541h55m56.99523286s) [upload=475239]: Upload finished. err: <nil> Duration: 30.003015573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............................X.....] duration: 30.106848751s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 30.106867105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX...] duration: 3.41257491s trace: https://jaeger.dev.storj.io/trace/2d0c9871e86208dd FAIL (2025-12-28 05:24:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.412604265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............................X....] duration: 44.066220555s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:24:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 44.066237769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.380259162s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:24:20 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.380288498s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X...] duration: 6.997384369s trace: https://jaeger.dev.storj.io/trace/3fb39e1edc5141d0 FAIL (2025-12-28 05:24:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:22:12.420613121 (541h53m12.291696156s) [upload=475201]: Starting upload Dec 28 05:22:12.420771243 (541h53m12.291854288s) [upload=475201]: Done starting upload Dec 28 05:22:12.420845324 (541h53m12.291928369s) [upload=475201]: Uploading segments... Dec 28 05:22:12.421160213 (541h53m12.292243259s) [upload=475201]: Got next segment. Inline: false Dec 28 05:22:12.421228063 (541h53m12.292311098s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:22:12.421235316 (541h53m12.292318361s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:22:12.667544147 (541h53m12.538627182s) [seg_pos={0 0}, upload=475201]: Begin upload segment... Dec 28 05:22:12.667600314 (541h53m12.538683359s) [seg_pos={0 0}, upload=475201]: Done begin upload segment. Dec 28 05:22:12.667651702 (541h53m12.538734737s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.667684795 (541h53m12.53876784s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668155992 (541h53m12.539239037s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668437407 (541h53m12.539520453s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668736868 (541h53m12.539819913s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669035085 (541h53m12.54011812s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669355186 (541h53m12.540438231s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669700623 (541h53m12.540783658s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.670116254 (541h53m12.541199289s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.671629636 (541h53m12.542712671s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.672775138 (541h53m12.543858173s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.674018926 (541h53m12.545101961s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.675494906 (541h53m12.546577941s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.677048325 (541h53m12.54813137s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.678590782 (541h53m12.549673837s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.680184747 (541h53m12.551267782s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:14.396211696 (541h53m14.267294731s) [upload=475201]: Next returned nil segment Dec 28 05:22:14.396227195 (541h53m14.26731023s) [upload=475201]: Waiting for error group managing segments... Dec 28 05:22:14.672101583 (541h53m14.543184618s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.683963370 (541h53m14.555046405s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.685615125 (541h53m14.55669817s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.692349790 (541h53m14.563432825s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.699057312 (541h53m14.570140347s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.702240734 (541h53m14.573323769s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.708070794 (541h53m14.57915384s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.881416702 (541h53m14.752499747s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.940713654 (541h53m14.811796699s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.035216782 (541h53m14.906299827s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.054383929 (541h53m14.925466974s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.055835914 (541h53m14.926918959s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652082331 (541h53m15.523165376s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652103341 (541h53m15.523186377s) [seg_pos={0 0}, upload=475201]: Segment reached optimal threshold of 13 pieces. Dec 28 05:22:15.652330624 (541h53m15.523413669s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652340343 (541h53m15.523423378s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652553719 (541h53m15.523636764s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652561233 (541h53m15.523644268s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652615297 (541h53m15.523698332s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652632459 (541h53m15.523715504s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652664780 (541h53m15.523747815s) [seg_pos={0 0}, upload=475201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:22:15.652818332 (541h53m15.523901367s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:22:15.652824285 (541h53m15.52390732s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:22:15.952630221 (541h53m15.823713266s) [upload=475201]: Done uploading segments... Dec 28 05:22:15.952656401 (541h53m15.823739447s) [upload=475201]: Upload finished. err: <nil> Duration: 6.997462045s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............................X...] duration: 42.934774323s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:24:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 42.934801876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [..X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX..] duration: 5.001035932s trace: https://jaeger.dev.storj.io/trace/2d0c9871e86208dd FAIL (2025-12-28 05:23:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001057513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X..] duration: 7.134704259s trace: https://jaeger.dev.storj.io/trace/3fb39e1edc5141d0 FAIL (2025-12-28 05:23:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:22:12.420613121 (541h53m12.291696156s) [upload=475201]: Starting upload Dec 28 05:22:12.420771243 (541h53m12.291854288s) [upload=475201]: Done starting upload Dec 28 05:22:12.420845324 (541h53m12.291928369s) [upload=475201]: Uploading segments... Dec 28 05:22:12.421160213 (541h53m12.292243259s) [upload=475201]: Got next segment. Inline: false Dec 28 05:22:12.421228063 (541h53m12.292311098s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:22:12.421235316 (541h53m12.292318361s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:22:12.667544147 (541h53m12.538627182s) [seg_pos={0 0}, upload=475201]: Begin upload segment... Dec 28 05:22:12.667600314 (541h53m12.538683359s) [seg_pos={0 0}, upload=475201]: Done begin upload segment. Dec 28 05:22:12.667651702 (541h53m12.538734737s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.667684795 (541h53m12.53876784s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668155992 (541h53m12.539239037s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668437407 (541h53m12.539520453s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668736868 (541h53m12.539819913s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669035085 (541h53m12.54011812s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669355186 (541h53m12.540438231s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669700623 (541h53m12.540783658s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.670116254 (541h53m12.541199289s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.671629636 (541h53m12.542712671s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.672775138 (541h53m12.543858173s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.674018926 (541h53m12.545101961s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.675494906 (541h53m12.546577941s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.677048325 (541h53m12.54813137s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.678590782 (541h53m12.549673837s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.680184747 (541h53m12.551267782s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:14.396211696 (541h53m14.267294731s) [upload=475201]: Next returned nil segment Dec 28 05:22:14.396227195 (541h53m14.26731023s) [upload=475201]: Waiting for error group managing segments... Dec 28 05:22:14.672101583 (541h53m14.543184618s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.683963370 (541h53m14.555046405s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.685615125 (541h53m14.55669817s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.692349790 (541h53m14.563432825s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.699057312 (541h53m14.570140347s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.702240734 (541h53m14.573323769s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.708070794 (541h53m14.57915384s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.881416702 (541h53m14.752499747s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.940713654 (541h53m14.811796699s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.035216782 (541h53m14.906299827s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.054383929 (541h53m14.925466974s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.055835914 (541h53m14.926918959s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652082331 (541h53m15.523165376s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652103341 (541h53m15.523186377s) [seg_pos={0 0}, upload=475201]: Segment reached optimal threshold of 13 pieces. Dec 28 05:22:15.652330624 (541h53m15.523413669s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652340343 (541h53m15.523423378s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652553719 (541h53m15.523636764s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652561233 (541h53m15.523644268s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652615297 (541h53m15.523698332s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652632459 (541h53m15.523715504s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652664780 (541h53m15.523747815s) [seg_pos={0 0}, upload=475201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:22:15.652818332 (541h53m15.523901367s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:22:15.652824285 (541h53m15.52390732s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:22:15.952630221 (541h53m15.823713266s) [upload=475201]: Done uploading segments... Dec 28 05:22:15.952656401 (541h53m15.823739447s) [upload=475201]: Upload finished. err: <nil> Duration: 7.134732321s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.943918231s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:23:15 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 4.943947536s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [...X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX.] duration: 5.000995524s trace: https://jaeger.dev.storj.io/trace/2d0c9871e86208dd FAIL (2025-12-28 05:22:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001022134s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X.] duration: 6.165030883s trace: https://jaeger.dev.storj.io/trace/3fb39e1edc5141d0 FAIL (2025-12-28 05:22:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:22:12.420613121 (541h53m12.291696156s) [upload=475201]: Starting upload Dec 28 05:22:12.420771243 (541h53m12.291854288s) [upload=475201]: Done starting upload Dec 28 05:22:12.420845324 (541h53m12.291928369s) [upload=475201]: Uploading segments... Dec 28 05:22:12.421160213 (541h53m12.292243259s) [upload=475201]: Got next segment. Inline: false Dec 28 05:22:12.421228063 (541h53m12.292311098s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:22:12.421235316 (541h53m12.292318361s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:22:12.667544147 (541h53m12.538627182s) [seg_pos={0 0}, upload=475201]: Begin upload segment... Dec 28 05:22:12.667600314 (541h53m12.538683359s) [seg_pos={0 0}, upload=475201]: Done begin upload segment. Dec 28 05:22:12.667651702 (541h53m12.538734737s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.667684795 (541h53m12.53876784s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668155992 (541h53m12.539239037s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668437407 (541h53m12.539520453s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668736868 (541h53m12.539819913s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669035085 (541h53m12.54011812s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669355186 (541h53m12.540438231s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669700623 (541h53m12.540783658s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.670116254 (541h53m12.541199289s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.671629636 (541h53m12.542712671s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.672775138 (541h53m12.543858173s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.674018926 (541h53m12.545101961s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.675494906 (541h53m12.546577941s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.677048325 (541h53m12.54813137s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.678590782 (541h53m12.549673837s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.680184747 (541h53m12.551267782s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:14.396211696 (541h53m14.267294731s) [upload=475201]: Next returned nil segment Dec 28 05:22:14.396227195 (541h53m14.26731023s) [upload=475201]: Waiting for error group managing segments... Dec 28 05:22:14.672101583 (541h53m14.543184618s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.683963370 (541h53m14.555046405s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.685615125 (541h53m14.55669817s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.692349790 (541h53m14.563432825s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.699057312 (541h53m14.570140347s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.702240734 (541h53m14.573323769s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.708070794 (541h53m14.57915384s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.881416702 (541h53m14.752499747s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.940713654 (541h53m14.811796699s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.035216782 (541h53m14.906299827s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.054383929 (541h53m14.925466974s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.055835914 (541h53m14.926918959s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652082331 (541h53m15.523165376s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652103341 (541h53m15.523186377s) [seg_pos={0 0}, upload=475201]: Segment reached optimal threshold of 13 pieces. Dec 28 05:22:15.652330624 (541h53m15.523413669s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652340343 (541h53m15.523423378s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652553719 (541h53m15.523636764s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652561233 (541h53m15.523644268s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652615297 (541h53m15.523698332s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652632459 (541h53m15.523715504s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652664780 (541h53m15.523747815s) [seg_pos={0 0}, upload=475201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:22:15.652818332 (541h53m15.523901367s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:22:15.652824285 (541h53m15.52390732s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:22:15.952630221 (541h53m15.823713266s) [upload=475201]: Done uploading segments... Dec 28 05:22:15.952656401 (541h53m15.823739447s) [upload=475201]: Upload finished. err: <nil> Duration: 6.165046762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.X] duration: 30.003130072s trace: https://jaeger.dev.storj.io/trace/3fb39e1edc5141d0 FAIL (2025-12-28 05:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 05:22:12.420613121 (541h53m12.291696156s) [upload=475201]: Starting upload Dec 28 05:22:12.420771243 (541h53m12.291854288s) [upload=475201]: Done starting upload Dec 28 05:22:12.420845324 (541h53m12.291928369s) [upload=475201]: Uploading segments... Dec 28 05:22:12.421160213 (541h53m12.292243259s) [upload=475201]: Got next segment. Inline: false Dec 28 05:22:12.421228063 (541h53m12.292311098s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:22:12.421235316 (541h53m12.292318361s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:22:12.667544147 (541h53m12.538627182s) [seg_pos={0 0}, upload=475201]: Begin upload segment... Dec 28 05:22:12.667600314 (541h53m12.538683359s) [seg_pos={0 0}, upload=475201]: Done begin upload segment. Dec 28 05:22:12.667651702 (541h53m12.538734737s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.667684795 (541h53m12.53876784s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668155992 (541h53m12.539239037s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668437407 (541h53m12.539520453s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.668736868 (541h53m12.539819913s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669035085 (541h53m12.54011812s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669355186 (541h53m12.540438231s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.669700623 (541h53m12.540783658s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.670116254 (541h53m12.541199289s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.671629636 (541h53m12.542712671s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.672775138 (541h53m12.543858173s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.674018926 (541h53m12.545101961s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.675494906 (541h53m12.546577941s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.677048325 (541h53m12.54813137s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.678590782 (541h53m12.549673837s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:12.680184747 (541h53m12.551267782s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Uploading piece... Dec 28 05:22:14.396211696 (541h53m14.267294731s) [upload=475201]: Next returned nil segment Dec 28 05:22:14.396227195 (541h53m14.26731023s) [upload=475201]: Waiting for error group managing segments... Dec 28 05:22:14.672101583 (541h53m14.543184618s) [noise=true , address=145.79.227.15:4873 , piece_id=WOYXY5FOKCQ4S6A6JDJVIRTHV2YBYR4544E7GYDGCOL3P72W27ZQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.683963370 (541h53m14.555046405s) [noise=true , address=145.79.227.38:4979 , piece_id=MS3VAJJY5QVABFCBQH3C7456JDAVLCEK2EC2R2FKZOE7P347V7LQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.685615125 (541h53m14.55669817s) [noise=true , address=145.79.227.7:4293 , piece_id=KDWF3S2RRDFUV3WWZ7QB7F3MLLWXCKZGHR3T5M6N7VJQDFMSUGBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.692349790 (541h53m14.563432825s) [noise=true , address=145.79.227.45:4230 , piece_id=Q3DRPYT4BW5S4VGXHKLYLF2BZLAK7QGMW6GGMYLLD5NVOJPPOW7A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.699057312 (541h53m14.570140347s) [noise=true , address=145.79.227.19:4079 , piece_id=76MHL5BUFR5IN3V3WULS7F5ZOAWX6DNTHXKKI7EYVOCU4EPXQYMQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.702240734 (541h53m14.573323769s) [noise=true , address=145.79.227.27:4691 , piece_id=4GUODLTZ4HD4Z7JIG76R2ESYAZERERKOM7SSHFXOXRKUZWBQ4Y5Q, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.708070794 (541h53m14.57915384s) [noise=true , address=145.79.227.46:4602 , piece_id=EITCPBNOTJV5GYVT6YL4MU4PIA3LWREQM4FJVJRIN7MDLXF6CRUQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.881416702 (541h53m14.752499747s) [noise=true , address=77.90.31.11:4556 , piece_id=CUA62ZMDHFONLOCUIN7EUV5AVJAAM2BXK4FPAWL3MND6PHNORIAA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:14.940713654 (541h53m14.811796699s) [noise=true , address=77.90.31.21:4877 , piece_id=YEAEAAKCYOSOSXCNO5QRGZR6RPG5F3L45V6WWN4SZJOYXVIYY5MQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.035216782 (541h53m14.906299827s) [noise=true , address=45.91.116.5:4297 , piece_id=XYFEIWKKKEONZUQYSN4HNRUUZGSEX53OE2FQVPC53GOGEX7U467A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.054383929 (541h53m14.925466974s) [noise=true , address=145.79.227.37:4565 , piece_id=YQ3N6FN44DIZ6YH75HCU4FXRAI6QBH4UMFTYS3LGIH5SL53I6WMA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.055835914 (541h53m14.926918959s) [noise=true , address=145.79.227.3:4150 , piece_id=L2ACR57C6HQ7JL6B6VIYRFOTKGVMHR6POVOGQ25HHVPZMKJYKF2A, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652082331 (541h53m15.523165376s) [noise=true , address=45.91.116.14:4189 , piece_id=A467S3HHUHRUJU4FCBEIEJPW343ZG2E54D7UE6RQNM2VXUV42UGA, seg_pos={0 0}, upload=475201]: Done uploading piece. err: <nil> Dec 28 05:22:15.652103341 (541h53m15.523186377s) [seg_pos={0 0}, upload=475201]: Segment reached optimal threshold of 13 pieces. Dec 28 05:22:15.652330624 (541h53m15.523413669s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652340343 (541h53m15.523423378s) [noise=true , address=45.91.116.8:4048 , piece_id=IWJETWY777OASO4P7UL3R3J2ZRAKG7KAASMBQRYJS2ATYSATZWLQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652553719 (541h53m15.523636764s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652561233 (541h53m15.523644268s) [noise=true , address=45.91.116.11:4120 , piece_id=WO3U6XKEMAJT2AV2YCMMFL2BEGIVXLXNP2AVAYALHCPPLHUTXBKQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652615297 (541h53m15.523698332s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 05:22:15.652632459 (541h53m15.523715504s) [noise=true , address=45.91.116.10:4084 , piece_id=PYNCI77LSSQKIYZLIFBSYCSGBD6LF2SA4DO37RDYHPKKZTXSCEBQ, seg_pos={0 0}, upload=475201]: Long tail cancellation due to optimal threshold Dec 28 05:22:15.652664780 (541h53m15.523747815s) [seg_pos={0 0}, upload=475201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:22:15.652818332 (541h53m15.523901367s) [upload=475201]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:22:15.652824285 (541h53m15.52390732s) [upload=475201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:22:15.952630221 (541h53m15.823713266s) [upload=475201]: Done uploading segments... Dec 28 05:22:15.952656401 (541h53m15.823739447s) [upload=475201]: Upload finished. err: <nil> Duration: 30.00315561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............................X..] duration: 16.282702973s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:22:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 16.282719776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............................X.] duration: 36.417499666s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:22:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 36.417528441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.253621305s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:22:01 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.253650541s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [....X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXXX] duration: 5.000647291s trace: https://jaeger.dev.storj.io/trace/2d0c9871e86208dd FAIL (2025-12-28 05:21:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000659314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.437082434s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:21:35 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.437117321s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............................X] duration: 1m0.003052483s trace: https://jaeger.dev.storj.io/trace/276f3da24cdc3bc FAIL (2025-12-28 05:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.077940953 (541h51m28.949023988s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078368778 (541h51m28.949451853s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078382754 (541h51m28.949465799s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078955746 (541h51m28.950038791s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.078967468 (541h51m28.950050503s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079384071 (541h51m28.950467106s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079390222 (541h51m28.950473267s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079440980 (541h51m28.950524025s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.079832234 (541h51m28.950915279s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080289806 (541h51m28.951372851s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.080788966 (541h51m28.951872011s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.081295871 (541h51m28.952378916s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.082079814 (541h51m28.953162859s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.083905420 (541h51m28.954988455s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:29.085563667 (541h51m28.956646702s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Uploading piece... Dec 28 05:20:30.383406637 (541h51m30.254489682s) [upload=475178]: Next returned nil segment Dec 28 05:20:30.383423890 (541h51m30.254507025s) [upload=475178]: Waiting for error group managing segments... Dec 28 05:20:30.714214738 (541h51m30.585297783s) [noise=true , address=64.49.13.26:4955 , piece_id=XDZY52ZS7R7MA5WHADVQYQCF4BEITL6DTK2XMDYA54TRJI74WVLQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.730365362 (541h51m30.601448407s) [noise=true , address=64.49.13.27:4965 , piece_id=GQOC27TRLRZ4AQZUQFZO5RTYQZ3BJDXOBEORNLPN5PFK3GZCISHQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:30.771353209 (541h51m30.642436244s) [noise=true , address=193.149.164.143:4695 , piece_id=RAJW7U23HZBRZGD2MIHUQTBCCT5HE2VMSZ7Z3ILJIATZUD6ULL3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.089396324 (541h51m30.960479369s) [noise=true , address=145.79.227.51:4418 , piece_id=BXHDUTB2DQHZDRC2X743QZR56U3MV275GGCEGY7WAJAWHDW56CPA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.147921396 (541h51m31.019004431s) [noise=true , address=77.90.31.13:4656 , piece_id=2QKAASYXRDXH47IB4YRFKZWSTFJXYR5TU7FGDFTKIVE44YBHQK3A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.148632219 (541h51m31.019715264s) [noise=true , address=77.90.31.21:4894 , piece_id=YRFDU2CVUF3ARE2ERED6636QX5E4Q2L2GYLMOWLCYQDLEOJULOBQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.154798240 (541h51m31.025881285s) [noise=true , address=77.90.31.11:4588 , piece_id=PW6P57MOIHXPYS5V34OZBFFK446X6QUEPAMMWHUAZPTPQ3QV46BA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.236517205 (541h51m31.10760025s) [noise=true , address=45.91.116.4:4944 , piece_id=J7OHZIWM25DFYIQ2RLQEDC6X6WM7VZJEDM67ILBMT6DFI7I36RAA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.241105242 (541h51m31.112188287s) [noise=true , address=77.90.31.14:4687 , piece_id=AATTELRZNKHYKN4PMRKZLPKBWDXY3FLFPDFVW54DPHDFHZ764NMA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.298209849 (541h51m31.169292884s) [noise=true , address=77.90.31.10:4550 , piece_id=TAKHGZMWNJE5KXTZISQGBQGOZI7YELVQ7HCGPJQR6IXHKWOWWY5A, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.336437591 (541h51m31.207520626s) [noise=true , address=77.90.31.18:4815 , piece_id=PMIK7G4WNKXCWKS2QR2CW6HWAR6ZGGCDT6OVUYEYGUINB5IO52ZQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:31.981555648 (541h51m31.852638693s) [noise=true , address=145.79.227.34:4909 , piece_id=PAA7MLXB4Z7UHEKGLF6QMG2DNVH3OYC5GPUY2T3P7UQM32WYA6PA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.038535366 (541h51m31.909618411s) [noise=true , address=145.79.227.30:4788 , piece_id=AJVK46RUXUTP5WTX7A3KWWUENHKGCXYKIMXSMVJWUYQWGWUNO2FA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145752946 (541h51m32.016835991s) [noise=true , address=145.79.227.40:4055 , piece_id=CGAKOH6MF466MFQU2475QKTS2EPLGMSBAU5P4CIMIOIDNTYZ57SA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: <nil> Dec 28 05:20:32.145790828 (541h51m32.016873873s) [seg_pos={0 0}, upload=475178]: Segment reached optimal threshold of 14 pieces. Dec 28 05:20:32.146039161 (541h51m32.017122196s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52050->64.49.13.169:4707: use of closed network connection Dec 28 05:20:32.146045252 (541h51m32.017128287s) [noise=true , address=64.49.13.169:4707 , piece_id=ELPTXBEPXRWO4OW2RV2ET326H3RNLBTRBG7YFYH2NIUAFB5ISVKA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146262335 (541h51m32.01734537s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146268698 (541h51m32.017351733s) [noise=true , address=145.79.227.44:4215 , piece_id=EK2M2GJMV2KZFELF3MGRVWCTOKU5CVIR7XWRNYTOE364VV75EWJA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146299496 (541h51m32.017382531s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:32.146339793 (541h51m32.017422838s) [noise=true , address=145.79.227.35:4485 , piece_id=LMBCUJDLBD46MIBBQ5WCPQHP4D4PUZFPQMV6DOUAM3ERLXSVYJQA, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.146490380 (541h51m32.017573415s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ): protocol: read tcp 188.166.225.170:34194->77.90.31.19:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:20:32.146497443 (541h51m32.017580488s) [noise=true , address=77.90.31.19:4861 , piece_id=6F2VBR2EOHJFLAU2TT6SRGLCRF6OQYTLGOAWBHIP5UIAOMQIFUDQ, seg_pos={0 0}, upload=475178]: Long tail cancellation due to optimal threshold Dec 28 05:20:32.147007324 (541h51m32.018090369s) [seg_pos={0 0}, upload=475178]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 05:20:32.147225509 (541h51m32.018308575s) [upload=475178]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:32.147236781 (541h51m32.018319816s) [upload=475178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:32.443794294 (541h51m32.314877339s) [upload=475178]: Done uploading segments... Dec 28 05:20:32.443813640 (541h51m32.314896676s) [upload=475178]: Upload finished. err: <nil> Duration: 1m0.00307662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX.] duration: 7.355950154s trace: https://jaeger.dev.storj.io/trace/74b2bd3dfc6aba5c FAIL (2025-12-28 05:21:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CEPTW5PQCNAZESCVLR4JRTC3M52AAGFCSA47AV5SW7LL3WNF3I6A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3DLS5MFNAOU6FEWZFLSEEWQVUASOK66TNVVTMPIVI2M25XIJNP6Q) context deadline exceeded ======LOGS====== gments... Dec 28 05:20:11.897737274 (541h51m11.768820309s) [upload=475172]: Got next segment. Inline: false Dec 28 05:20:11.897792309 (541h51m11.768875354s) [upload=475172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:20:11.897844157 (541h51m11.768927202s) [upload=475172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:20:12.145279845 (541h51m12.01636289s) [seg_pos={0 0}, upload=475172]: Begin upload segment... Dec 28 05:20:12.145378883 (541h51m12.016461928s) [seg_pos={0 0}, upload=475172]: Done begin upload segment. Dec 28 05:20:12.145424029 (541h51m12.016507064s) [noise=true , address=145.79.227.45:4229 , piece_id=KRJJTZWQ2CVKWXAIWIYJPLZRAVTNW4LAJVEX33QJTCPTO6SDYUIA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.146456125 (541h51m12.01753917s) [noise=true , address=45.91.116.10:4082 , piece_id=3DLS5MFNAOU6FEWZFLSEEWQVUASOK66TNVVTMPIVI2M25XIJNP6Q, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.147576368 (541h51m12.018659403s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7AVNBMQR57T6SCEIM6Y23GSLQBNZOFWI4CJ2JPPXOK2LUDI6KCQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.148118170 (541h51m12.019201215s) [noise=true , address=145.79.227.19:4079 , piece_id=QJR5JIECYNAFJPJSMMNC3D6LPH3A76VZ3UIWXN76YTLT5JPE5FNQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.148766644 (541h51m12.019849689s) [noise=true , address=145.79.227.51:4410 , piece_id=3HBXRUSYQGOMBOYZDEUDDEYO6VG6755ZM2PNLUDU7W5JHNVC4GBA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.149231480 (541h51m12.020314525s) [noise=true , address=145.79.227.14:4837 , piece_id=N7IQEQU47INPTOOH3J36PCNGMLMWJW7YVNAUM3YPF3X6LVDVRKOQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.149681747 (541h51m12.020764782s) [noise=true , address=77.90.31.14:4662 , piece_id=CEPTW5PQCNAZESCVLR4JRTC3M52AAGFCSA47AV5SW7LL3WNF3I6A, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.150164966 (541h51m12.021248012s) [noise=true , address=145.79.227.38:4977 , piece_id=WLR2KETVVZMVPHACSKBTHYDKS6TEV4KHLDSLLEBMK4VNKYMQCE6A, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.150923040 (541h51m12.022006085s) [noise=true , address=145.79.227.47:4267 , piece_id=JEIUE6PGT244MS63KHNHVVA4WY4IKMDKUGLE5SJWPAB3W3XIHDIQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.151749454 (541h51m12.022832499s) [noise=true , address=145.79.227.42:4122 , piece_id=HSXAM3W5YWI74VLUOKQUTOC3WKJX5Q25LVLNTIGBOZX4EVJ2F7SQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.152260067 (541h51m12.023343112s) [noise=true , address=145.79.227.15:4873 , piece_id=H5EAKPNEXFGNV4RAONEGRU3T4LIN7H4TUUWHXVTLVQ6J7HUMF4OA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.152957475 (541h51m12.02404052s) [noise=true , address=145.79.227.2:4115 , piece_id=EBS7YETIWBYK265WB73YM3QN5VRR5A4IZZFMBWRWDXXWFA6W7OMA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.153564700 (541h51m12.024647746s) [noise=true , address=145.79.227.24:4583 , piece_id=HF5TDUWLHDR2WWLKEEOYVOJDDF3BTPLXOJSYCTS6NDHZPEUABEGA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.154107815 (541h51m12.02519086s) [noise=true , address=145.79.227.7:4293 , piece_id=XULEH7TAUNTSIQ4WLHK7U7DRMZMP6GIJLTOEYZ3CKSLFIEZBYNSQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.154758674 (541h51m12.025841729s) [noise=true , address=45.91.116.7:4009 , piece_id=PDQRR3V27XHWSYS5PJH2UBCWSWMOYRFOB6VLHCSO5PQEZO7MLAJQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.155337596 (541h51m12.026420641s) [noise=true , address=145.79.227.13:4802 , piece_id=DS2ZUIHS2RMW4E2DON7NPPTVEQCTAWZYSWEDPFTJIHF2FTTK4WXA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:13.958641065 (541h51m13.82972411s) [upload=475172]: Next returned nil segment Dec 28 05:20:13.958655602 (541h51m13.829738637s) [upload=475172]: Waiting for error group managing segments... Dec 28 05:20:14.291420119 (541h51m14.162503174s) [noise=true , address=145.79.227.13:4802 , piece_id=DS2ZUIHS2RMW4E2DON7NPPTVEQCTAWZYSWEDPFTJIHF2FTTK4WXA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.295566294 (541h51m14.166649329s) [noise=true , address=145.79.227.14:4837 , piece_id=N7IQEQU47INPTOOH3J36PCNGMLMWJW7YVNAUM3YPF3X6LVDVRKOQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.314352747 (541h51m14.185435792s) [noise=true , address=145.79.227.2:4115 , piece_id=EBS7YETIWBYK265WB73YM3QN5VRR5A4IZZFMBWRWDXXWFA6W7OMA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.315025187 (541h51m14.186108222s) [noise=true , address=145.79.227.15:4873 , piece_id=H5EAKPNEXFGNV4RAONEGRU3T4LIN7H4TUUWHXVTLVQ6J7HUMF4OA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.317863651 (541h51m14.188946696s) [noise=true , address=145.79.227.51:4410 , piece_id=3HBXRUSYQGOMBOYZDEUDDEYO6VG6755ZM2PNLUDU7W5JHNVC4GBA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.318268693 (541h51m14.189351728s) [noise=true , address=145.79.227.24:4583 , piece_id=HF5TDUWLHDR2WWLKEEOYVOJDDF3BTPLXOJSYCTS6NDHZPEUABEGA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.322109776 (541h51m14.193192821s) [noise=true , address=77.90.31.14:4662 , piece_id=CEPTW5PQCNAZESCVLR4JRTC3M52AAGFCSA47AV5SW7LL3WNF3I6A, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.324353349 (541h51m14.195436394s) [noise=true , address=145.79.227.45:4229 , piece_id=KRJJTZWQ2CVKWXAIWIYJPLZRAVTNW4LAJVEX33QJTCPTO6SDYUIA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.324826641 (541h51m14.195909686s) [noise=true , address=145.79.227.38:4977 , piece_id=WLR2KETVVZMVPHACSKBTHYDKS6TEV4KHLDSLLEBMK4VNKYMQCE6A, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.327273971 (541h51m14.198357026s) [noise=true , address=145.79.227.47:4267 , piece_id=JEIUE6PGT244MS63KHNHVVA4WY4IKMDKUGLE5SJWPAB3W3XIHDIQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.344381265 (541h51m14.21546431s) [noise=true , address=145.79.227.19:4079 , piece_id=QJR5JIECYNAFJPJSMMNC3D6LPH3A76VZ3UIWXN76YTLT5JPE5FNQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.489792541 (541h51m14.360875586s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7AVNBMQR57T6SCEIM6Y23GSLQBNZOFWI4CJ2JPPXOK2LUDI6KCQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.559342015 (541h51m14.43042506s) [noise=true , address=45.91.116.10:4082 , piece_id=3DLS5MFNAOU6FEWZFLSEEWQVUASOK66TNVVTMPIVI2M25XIJNP6Q, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.559373615 (541h51m14.43045666s) [seg_pos={0 0}, upload=475172]: Segment reached optimal threshold of 13 pieces. Dec 28 05:20:14.559679658 (541h51m14.430762713s) [noise=true , address=45.91.116.7:4009 , piece_id=PDQRR3V27XHWSYS5PJH2UBCWSWMOYRFOB6VLHCSO5PQEZO7MLAJQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:46152->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:46152->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46152->45.91.116.7:4009: use of closed network connection Dec 28 05:20:14.559689266 (541h51m14.430772301s) [noise=true , address=45.91.116.7:4009 , piece_id=PDQRR3V27XHWSYS5PJH2UBCWSWMOYRFOB6VLHCSO5PQEZO7MLAJQ, seg_pos={0 0}, upload=475172]: Long tail cancellation due to optimal threshold Dec 28 05:20:14.559768568 (541h51m14.430851603s) [noise=true , address=145.79.227.7:4293 , piece_id=XULEH7TAUNTSIQ4WLHK7U7DRMZMP6GIJLTOEYZ3CKSLFIEZBYNSQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:14.559785880 (541h51m14.430868915s) [noise=true , address=145.79.227.7:4293 , piece_id=XULEH7TAUNTSIQ4WLHK7U7DRMZMP6GIJLTOEYZ3CKSLFIEZBYNSQ, seg_pos={0 0}, upload=475172]: Long tail cancellation due to optimal threshold Dec 28 05:20:14.560011520 (541h51m14.431094565s) [noise=true , address=145.79.227.42:4122 , piece_id=HSXAM3W5YWI74VLUOKQUTOC3WKJX5Q25LVLNTIGBOZX4EVJ2F7SQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:48630->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:48630->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48630->145.79.227.42:4122: use of closed network connection Dec 28 05:20:14.560019996 (541h51m14.431103061s) [noise=true , address=145.79.227.42:4122 , piece_id=HSXAM3W5YWI74VLUOKQUTOC3WKJX5Q25LVLNTIGBOZX4EVJ2F7SQ, seg_pos={0 0}, upload=475172]: Long tail cancellation due to optimal threshold Dec 28 05:20:14.560052188 (541h51m14.431135233s) [seg_pos={0 0}, upload=475172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:20:14.560146207 (541h51m14.431229252s) [upload=475172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:14.560153320 (541h51m14.431236365s) [upload=475172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:14.857195546 (541h51m14.728278581s) [upload=475172]: Done uploading segments... Dec 28 05:20:14.857215284 (541h51m14.728298329s) [upload=475172]: Upload finished. err: <nil> Duration: 7.355983978s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.666644916s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:21:00 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.666674282s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.....X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XXX] duration: 5.001978776s trace: https://jaeger.dev.storj.io/trace/2f3eed1ea8150256 FAIL (2025-12-28 05:20:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00200143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..XX] duration: 30.003971576s trace: https://jaeger.dev.storj.io/trace/74b2bd3dfc6aba5c FAIL (2025-12-28 05:20:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CEPTW5PQCNAZESCVLR4JRTC3M52AAGFCSA47AV5SW7LL3WNF3I6A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 3DLS5MFNAOU6FEWZFLSEEWQVUASOK66TNVVTMPIVI2M25XIJNP6Q) context deadline exceeded ======LOGS====== gments... Dec 28 05:20:11.897737274 (541h51m11.768820309s) [upload=475172]: Got next segment. Inline: false Dec 28 05:20:11.897792309 (541h51m11.768875354s) [upload=475172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:20:11.897844157 (541h51m11.768927202s) [upload=475172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:20:12.145279845 (541h51m12.01636289s) [seg_pos={0 0}, upload=475172]: Begin upload segment... Dec 28 05:20:12.145378883 (541h51m12.016461928s) [seg_pos={0 0}, upload=475172]: Done begin upload segment. Dec 28 05:20:12.145424029 (541h51m12.016507064s) [noise=true , address=145.79.227.45:4229 , piece_id=KRJJTZWQ2CVKWXAIWIYJPLZRAVTNW4LAJVEX33QJTCPTO6SDYUIA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.146456125 (541h51m12.01753917s) [noise=true , address=45.91.116.10:4082 , piece_id=3DLS5MFNAOU6FEWZFLSEEWQVUASOK66TNVVTMPIVI2M25XIJNP6Q, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.147576368 (541h51m12.018659403s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7AVNBMQR57T6SCEIM6Y23GSLQBNZOFWI4CJ2JPPXOK2LUDI6KCQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.148118170 (541h51m12.019201215s) [noise=true , address=145.79.227.19:4079 , piece_id=QJR5JIECYNAFJPJSMMNC3D6LPH3A76VZ3UIWXN76YTLT5JPE5FNQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.148766644 (541h51m12.019849689s) [noise=true , address=145.79.227.51:4410 , piece_id=3HBXRUSYQGOMBOYZDEUDDEYO6VG6755ZM2PNLUDU7W5JHNVC4GBA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.149231480 (541h51m12.020314525s) [noise=true , address=145.79.227.14:4837 , piece_id=N7IQEQU47INPTOOH3J36PCNGMLMWJW7YVNAUM3YPF3X6LVDVRKOQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.149681747 (541h51m12.020764782s) [noise=true , address=77.90.31.14:4662 , piece_id=CEPTW5PQCNAZESCVLR4JRTC3M52AAGFCSA47AV5SW7LL3WNF3I6A, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.150164966 (541h51m12.021248012s) [noise=true , address=145.79.227.38:4977 , piece_id=WLR2KETVVZMVPHACSKBTHYDKS6TEV4KHLDSLLEBMK4VNKYMQCE6A, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.150923040 (541h51m12.022006085s) [noise=true , address=145.79.227.47:4267 , piece_id=JEIUE6PGT244MS63KHNHVVA4WY4IKMDKUGLE5SJWPAB3W3XIHDIQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.151749454 (541h51m12.022832499s) [noise=true , address=145.79.227.42:4122 , piece_id=HSXAM3W5YWI74VLUOKQUTOC3WKJX5Q25LVLNTIGBOZX4EVJ2F7SQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.152260067 (541h51m12.023343112s) [noise=true , address=145.79.227.15:4873 , piece_id=H5EAKPNEXFGNV4RAONEGRU3T4LIN7H4TUUWHXVTLVQ6J7HUMF4OA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.152957475 (541h51m12.02404052s) [noise=true , address=145.79.227.2:4115 , piece_id=EBS7YETIWBYK265WB73YM3QN5VRR5A4IZZFMBWRWDXXWFA6W7OMA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.153564700 (541h51m12.024647746s) [noise=true , address=145.79.227.24:4583 , piece_id=HF5TDUWLHDR2WWLKEEOYVOJDDF3BTPLXOJSYCTS6NDHZPEUABEGA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.154107815 (541h51m12.02519086s) [noise=true , address=145.79.227.7:4293 , piece_id=XULEH7TAUNTSIQ4WLHK7U7DRMZMP6GIJLTOEYZ3CKSLFIEZBYNSQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.154758674 (541h51m12.025841729s) [noise=true , address=45.91.116.7:4009 , piece_id=PDQRR3V27XHWSYS5PJH2UBCWSWMOYRFOB6VLHCSO5PQEZO7MLAJQ, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:12.155337596 (541h51m12.026420641s) [noise=true , address=145.79.227.13:4802 , piece_id=DS2ZUIHS2RMW4E2DON7NPPTVEQCTAWZYSWEDPFTJIHF2FTTK4WXA, seg_pos={0 0}, upload=475172]: Uploading piece... Dec 28 05:20:13.958641065 (541h51m13.82972411s) [upload=475172]: Next returned nil segment Dec 28 05:20:13.958655602 (541h51m13.829738637s) [upload=475172]: Waiting for error group managing segments... Dec 28 05:20:14.291420119 (541h51m14.162503174s) [noise=true , address=145.79.227.13:4802 , piece_id=DS2ZUIHS2RMW4E2DON7NPPTVEQCTAWZYSWEDPFTJIHF2FTTK4WXA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.295566294 (541h51m14.166649329s) [noise=true , address=145.79.227.14:4837 , piece_id=N7IQEQU47INPTOOH3J36PCNGMLMWJW7YVNAUM3YPF3X6LVDVRKOQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.314352747 (541h51m14.185435792s) [noise=true , address=145.79.227.2:4115 , piece_id=EBS7YETIWBYK265WB73YM3QN5VRR5A4IZZFMBWRWDXXWFA6W7OMA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.315025187 (541h51m14.186108222s) [noise=true , address=145.79.227.15:4873 , piece_id=H5EAKPNEXFGNV4RAONEGRU3T4LIN7H4TUUWHXVTLVQ6J7HUMF4OA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.317863651 (541h51m14.188946696s) [noise=true , address=145.79.227.51:4410 , piece_id=3HBXRUSYQGOMBOYZDEUDDEYO6VG6755ZM2PNLUDU7W5JHNVC4GBA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.318268693 (541h51m14.189351728s) [noise=true , address=145.79.227.24:4583 , piece_id=HF5TDUWLHDR2WWLKEEOYVOJDDF3BTPLXOJSYCTS6NDHZPEUABEGA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.322109776 (541h51m14.193192821s) [noise=true , address=77.90.31.14:4662 , piece_id=CEPTW5PQCNAZESCVLR4JRTC3M52AAGFCSA47AV5SW7LL3WNF3I6A, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.324353349 (541h51m14.195436394s) [noise=true , address=145.79.227.45:4229 , piece_id=KRJJTZWQ2CVKWXAIWIYJPLZRAVTNW4LAJVEX33QJTCPTO6SDYUIA, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.324826641 (541h51m14.195909686s) [noise=true , address=145.79.227.38:4977 , piece_id=WLR2KETVVZMVPHACSKBTHYDKS6TEV4KHLDSLLEBMK4VNKYMQCE6A, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.327273971 (541h51m14.198357026s) [noise=true , address=145.79.227.47:4267 , piece_id=JEIUE6PGT244MS63KHNHVVA4WY4IKMDKUGLE5SJWPAB3W3XIHDIQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.344381265 (541h51m14.21546431s) [noise=true , address=145.79.227.19:4079 , piece_id=QJR5JIECYNAFJPJSMMNC3D6LPH3A76VZ3UIWXN76YTLT5JPE5FNQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.489792541 (541h51m14.360875586s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7AVNBMQR57T6SCEIM6Y23GSLQBNZOFWI4CJ2JPPXOK2LUDI6KCQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.559342015 (541h51m14.43042506s) [noise=true , address=45.91.116.10:4082 , piece_id=3DLS5MFNAOU6FEWZFLSEEWQVUASOK66TNVVTMPIVI2M25XIJNP6Q, seg_pos={0 0}, upload=475172]: Done uploading piece. err: <nil> Dec 28 05:20:14.559373615 (541h51m14.43045666s) [seg_pos={0 0}, upload=475172]: Segment reached optimal threshold of 13 pieces. Dec 28 05:20:14.559679658 (541h51m14.430762713s) [noise=true , address=45.91.116.7:4009 , piece_id=PDQRR3V27XHWSYS5PJH2UBCWSWMOYRFOB6VLHCSO5PQEZO7MLAJQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:46152->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:46152->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46152->45.91.116.7:4009: use of closed network connection Dec 28 05:20:14.559689266 (541h51m14.430772301s) [noise=true , address=45.91.116.7:4009 , piece_id=PDQRR3V27XHWSYS5PJH2UBCWSWMOYRFOB6VLHCSO5PQEZO7MLAJQ, seg_pos={0 0}, upload=475172]: Long tail cancellation due to optimal threshold Dec 28 05:20:14.559768568 (541h51m14.430851603s) [noise=true , address=145.79.227.7:4293 , piece_id=XULEH7TAUNTSIQ4WLHK7U7DRMZMP6GIJLTOEYZ3CKSLFIEZBYNSQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: expected piece hash; context canceled; EOF Dec 28 05:20:14.559785880 (541h51m14.430868915s) [noise=true , address=145.79.227.7:4293 , piece_id=XULEH7TAUNTSIQ4WLHK7U7DRMZMP6GIJLTOEYZ3CKSLFIEZBYNSQ, seg_pos={0 0}, upload=475172]: Long tail cancellation due to optimal threshold Dec 28 05:20:14.560011520 (541h51m14.431094565s) [noise=true , address=145.79.227.42:4122 , piece_id=HSXAM3W5YWI74VLUOKQUTOC3WKJX5Q25LVLNTIGBOZX4EVJ2F7SQ, seg_pos={0 0}, upload=475172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:48630->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:48630->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48630->145.79.227.42:4122: use of closed network connection Dec 28 05:20:14.560019996 (541h51m14.431103061s) [noise=true , address=145.79.227.42:4122 , piece_id=HSXAM3W5YWI74VLUOKQUTOC3WKJX5Q25LVLNTIGBOZX4EVJ2F7SQ, seg_pos={0 0}, upload=475172]: Long tail cancellation due to optimal threshold Dec 28 05:20:14.560052188 (541h51m14.431135233s) [seg_pos={0 0}, upload=475172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:20:14.560146207 (541h51m14.431229252s) [upload=475172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:20:14.560153320 (541h51m14.431236365s) [upload=475172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:20:14.857195546 (541h51m14.728278581s) [upload=475172]: Done uploading segments... Dec 28 05:20:14.857215284 (541h51m14.728298329s) [upload=475172]: Upload finished. err: <nil> Duration: 30.003999449s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.179803331s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:20:22 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.179857144s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 43.027121531s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:20:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 43.027151397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..X] duration: 30.005747479s trace: https://jaeger.dev.storj.io/trace/249363892283bba8 FAIL (2025-12-28 05:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: WSVD7YVUEQCCITLFJPDHRR73ZAQUESENK4TJEWTUSJU7BB4NXZAA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 3G2ICA636YGWFORPKOGDXGTHFS2RAYGY63K5XSAN6JFAWCWBM6BQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: ILQSRKUNTX2LKW7ZSLAGC7ZKPFEIHGPAYORKGM4WULXD7UJ42XRQ) context deadline exceeded ======LOGS====== Dec 28 05:19:32.475646628 (541h50m32.346729693s) [upload=475161]: Starting upload Dec 28 05:19:32.475750856 (541h50m32.346833912s) [upload=475161]: Done starting upload Dec 28 05:19:32.475852179 (541h50m32.346935215s) [upload=475161]: Uploading segments... Dec 28 05:19:32.475984071 (541h50m32.347067106s) [upload=475161]: Got next segment. Inline: false Dec 28 05:19:32.476013327 (541h50m32.347096372s) [upload=475161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:19:32.476018967 (541h50m32.347102002s) [upload=475161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:19:32.721462171 (541h50m32.592545206s) [seg_pos={0 0}, upload=475161]: Begin upload segment... Dec 28 05:19:32.721524520 (541h50m32.592607565s) [seg_pos={0 0}, upload=475161]: Done begin upload segment. Dec 28 05:19:32.721590896 (541h50m32.592673931s) [noise=true , address=145.79.227.52:4445 , piece_id=LCSF4L2ZLE2PBTYPWOQXVBHZAWBLDM3TRDQYEYXA2ZZR6VYZSEKQ, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.721632595 (541h50m32.59271563s) [noise=true , address=145.79.227.25:4617 , piece_id=ZMI7MHQXJQF2GV5ZTDFOZLB6MQBZI2ZZD5YYC3HNWRUZPFEL6QUA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.722206067 (541h50m32.593289102s) [noise=true , address=145.79.227.9:4365 , piece_id=2NS3MJG6DBIHUHOTHNQXZSHZD6O4EQEZIVJTFRU4IICXWF7PZDGA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.722220355 (541h50m32.59330339s) [noise=true , address=45.91.116.5:4298 , piece_id=LGSPWNLQZBFEPIYZLW3T6A2E67DOWL27EZDPZZ3NOLMQW6BBYODA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.722794958 (541h50m32.593878003s) [noise=true , address=77.90.31.18:4805 , piece_id=ADS7FR3BTT3V6CAMJF5BVFFIXDV5QPCQW2DW4ZIRDFK6WGUYI57A, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.722829564 (541h50m32.593912599s) [noise=true , address=145.79.227.36:4517 , piece_id=BJ6YJBJFFXEPXUB3BPW4GVIG7OFPK43CQKWUS2XEGIVNMLBBK3IQ, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.723285693 (541h50m32.594368728s) [noise=true , address=77.90.31.10:4519 , piece_id=XGUW7OWRHEDJUGJIHJKU4TILMCX5ZX57YKGWFBM2LDTMLTXLVUDA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.723732153 (541h50m32.594815188s) [noise=true , address=145.79.227.15:4876 , piece_id=M2DXGPC3PBASVHNJLES2D63Z32476U5HDAB4CF2QN63PEPNQHODA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.724146332 (541h50m32.595229367s) [noise=true , address=145.79.227.5:4222 , piece_id=P3UEBKKBQPATXI2SZBD5R4XHBHWDTZUFJ477Y5FWQM5FYINV5XYQ, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.724744050 (541h50m32.595827095s) [noise=true , address=145.79.227.39:4015 , piece_id=C5K4ZEAOGNE2LCGBAV3V3AT5UOJCU6HW6BLTZKRQF3YUVJ42AZBA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.725964964 (541h50m32.597048s) [noise=true , address=145.79.227.2:4114 , piece_id=ZYTE5AHN4EM42E5TVZQWW776X52XFGJ2FEFGJ3OLV3NVSZODC5TQ, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.726417807 (541h50m32.597500852s) [noise=true , address=145.79.227.11:4732 , piece_id=ILJKDTVPQPZVM2IUJ2WJBQTIFMFGHX3FBVYJ76MVKZ2B4QYK2SRA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.726993103 (541h50m32.598076148s) [noise=true , address=145.79.227.41:4086 , piece_id=XMSVLUCEXCNL4D325H3K6ERSUAFANMTTE75EIAT574Q2V2YF7C7A, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.727475310 (541h50m32.598558355s) [noise=true , address=77.90.31.19:4842 , piece_id=WSVD7YVUEQCCITLFJPDHRR73ZAQUESENK4TJEWTUSJU7BB4NXZAA, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.727688507 (541h50m32.598771542s) [noise=true , address=45.91.116.7:4009 , piece_id=3G2ICA636YGWFORPKOGDXGTHFS2RAYGY63K5XSAN6JFAWCWBM6BQ, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:32.728558524 (541h50m32.599641559s) [noise=true , address=45.91.116.8:4047 , piece_id=ILQSRKUNTX2LKW7ZSLAGC7ZKPFEIHGPAYORKGM4WULXD7UJ42XRQ, seg_pos={0 0}, upload=475161]: Uploading piece... Dec 28 05:19:34.450654630 (541h50m34.321737675s) [upload=475161]: Next returned nil segment Dec 28 05:19:34.450685208 (541h50m34.321768263s) [upload=475161]: Waiting for error group managing segments... Dec 28 05:19:34.730226610 (541h50m34.601309645s) [noise=true , address=145.79.227.15:4876 , piece_id=M2DXGPC3PBASVHNJLES2D63Z32476U5HDAB4CF2QN63PEPNQHODA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.731748457 (541h50m34.602831492s) [noise=true , address=145.79.227.11:4732 , piece_id=ILJKDTVPQPZVM2IUJ2WJBQTIFMFGHX3FBVYJ76MVKZ2B4QYK2SRA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.738912298 (541h50m34.609995343s) [noise=true , address=145.79.227.36:4517 , piece_id=BJ6YJBJFFXEPXUB3BPW4GVIG7OFPK43CQKWUS2XEGIVNMLBBK3IQ, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.742521390 (541h50m34.613604425s) [noise=true , address=145.79.227.41:4086 , piece_id=XMSVLUCEXCNL4D325H3K6ERSUAFANMTTE75EIAT574Q2V2YF7C7A, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.749032508 (541h50m34.620115543s) [noise=true , address=145.79.227.52:4445 , piece_id=LCSF4L2ZLE2PBTYPWOQXVBHZAWBLDM3TRDQYEYXA2ZZR6VYZSEKQ, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.756362095 (541h50m34.62744513s) [noise=true , address=145.79.227.25:4617 , piece_id=ZMI7MHQXJQF2GV5ZTDFOZLB6MQBZI2ZZD5YYC3HNWRUZPFEL6QUA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.757489191 (541h50m34.628572226s) [noise=true , address=145.79.227.9:4365 , piece_id=2NS3MJG6DBIHUHOTHNQXZSHZD6O4EQEZIVJTFRU4IICXWF7PZDGA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.762535681 (541h50m34.633618726s) [noise=true , address=145.79.227.2:4114 , piece_id=ZYTE5AHN4EM42E5TVZQWW776X52XFGJ2FEFGJ3OLV3NVSZODC5TQ, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.763066342 (541h50m34.634149387s) [noise=true , address=145.79.227.39:4015 , piece_id=C5K4ZEAOGNE2LCGBAV3V3AT5UOJCU6HW6BLTZKRQF3YUVJ42AZBA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.763278466 (541h50m34.634361501s) [noise=true , address=145.79.227.5:4222 , piece_id=P3UEBKKBQPATXI2SZBD5R4XHBHWDTZUFJ477Y5FWQM5FYINV5XYQ, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.912505647 (541h50m34.783588682s) [noise=true , address=77.90.31.19:4842 , piece_id=WSVD7YVUEQCCITLFJPDHRR73ZAQUESENK4TJEWTUSJU7BB4NXZAA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.959516353 (541h50m34.830599398s) [noise=true , address=45.91.116.8:4047 , piece_id=ILQSRKUNTX2LKW7ZSLAGC7ZKPFEIHGPAYORKGM4WULXD7UJ42XRQ, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.965087312 (541h50m34.836170347s) [noise=true , address=45.91.116.7:4009 , piece_id=3G2ICA636YGWFORPKOGDXGTHFS2RAYGY63K5XSAN6JFAWCWBM6BQ, seg_pos={0 0}, upload=475161]: Done uploading piece. err: <nil> Dec 28 05:19:34.965110786 (541h50m34.836193831s) [seg_pos={0 0}, upload=475161]: Segment reached optimal threshold of 13 pieces. Dec 28 05:19:34.965345293 (541h50m34.836428328s) [noise=true , address=77.90.31.18:4805 , piece_id=ADS7FR3BTT3V6CAMJF5BVFFIXDV5QPCQW2DW4ZIRDFK6WGUYI57A, seg_pos={0 0}, upload=475161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 05:19:34.965354190 (541h50m34.836437225s) [noise=true , address=77.90.31.18:4805 , piece_id=ADS7FR3BTT3V6CAMJF5BVFFIXDV5QPCQW2DW4ZIRDFK6WGUYI57A, seg_pos={0 0}, upload=475161]: Long tail cancellation due to optimal threshold Dec 28 05:19:34.965673067 (541h50m34.836756112s) [noise=true , address=77.90.31.10:4519 , piece_id=XGUW7OWRHEDJUGJIHJKU4TILMCX5ZX57YKGWFBM2LDTMLTXLVUDA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:32980->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:19:34.965683157 (541h50m34.836766192s) [noise=true , address=77.90.31.10:4519 , piece_id=XGUW7OWRHEDJUGJIHJKU4TILMCX5ZX57YKGWFBM2LDTMLTXLVUDA, seg_pos={0 0}, upload=475161]: Long tail cancellation due to optimal threshold Dec 28 05:19:34.965761977 (541h50m34.836845012s) [noise=true , address=45.91.116.5:4298 , piece_id=LGSPWNLQZBFEPIYZLW3T6A2E67DOWL27EZDPZZ3NOLMQW6BBYODA, seg_pos={0 0}, upload=475161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 05:19:34.965818815 (541h50m34.83690185s) [noise=true , address=45.91.116.5:4298 , piece_id=LGSPWNLQZBFEPIYZLW3T6A2E67DOWL27EZDPZZ3NOLMQW6BBYODA, seg_pos={0 0}, upload=475161]: Long tail cancellation due to optimal threshold Dec 28 05:19:34.965841828 (541h50m34.836924863s) [seg_pos={0 0}, upload=475161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:19:34.965942951 (541h50m34.837025996s) [upload=475161]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:19:34.965948271 (541h50m34.837031306s) [upload=475161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:19:35.258422880 (541h50m35.129505925s) [upload=475161]: Done uploading segments... Dec 28 05:19:35.258443890 (541h50m35.129526925s) [upload=475161]: Upload finished. err: <nil> Duration: 30.005778527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [......X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....XX] duration: 5.001536356s trace: https://jaeger.dev.storj.io/trace/23ead6b616b67dfa FAIL (2025-12-28 05:19:49 UTC) orbiter warning: context deadline exceeded Duration: 5.001561614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.459533354s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:19:24 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.459561717s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.......X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.....XXX....X] duration: 5.001232775s trace: https://jaeger.dev.storj.io/trace/6a171383ed3f2a0 FAIL (2025-12-28 05:19:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001256551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 45.610120704s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:19:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 45.610162845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.XX.X.XX..] duration: 5.952920841s trace: https://jaeger.dev.storj.io/trace/3eab9a4204e3aeb8 FAIL (2025-12-28 05:18:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 5OLALQWBBQLJUNAWOASIHPYAYZ4OEPSZXVBJUPCFAEUIWGPAEWFA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CW5X37NUMNFTMMP6FIQKNRNZD5RBNM547GCKOBDGAXQ7JKN2CECQ) context deadline exceeded ======LOGS====== Dec 28 05:16:55.194888042 (541h47m55.065971087s) [upload=475126]: Starting upload Dec 28 05:16:55.194946384 (541h47m55.066029429s) [upload=475126]: Done starting upload Dec 28 05:16:55.195038490 (541h47m55.066121535s) [upload=475126]: Uploading segments... Dec 28 05:16:55.195200447 (541h47m55.066283482s) [upload=475126]: Got next segment. Inline: false Dec 28 05:16:55.195262746 (541h47m55.066345791s) [upload=475126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:16:55.195272915 (541h47m55.06635596s) [upload=475126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:16:55.440289767 (541h47m55.311372812s) [seg_pos={0 0}, upload=475126]: Begin upload segment... Dec 28 05:16:55.440341306 (541h47m55.311424341s) [seg_pos={0 0}, upload=475126]: Done begin upload segment. Dec 28 05:16:55.440368498 (541h47m55.311451533s) [noise=true , address=45.91.116.7:4010 , piece_id=CW5X37NUMNFTMMP6FIQKNRNZD5RBNM547GCKOBDGAXQ7JKN2CECQ, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.440517732 (541h47m55.311600777s) [noise=true , address=145.79.227.46:4603 , piece_id=447I7VFIUL42O7ZLM2LBCTERPXZDAVNHSLEF2TAIYYRHJF2TNK5Q, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.441160465 (541h47m55.31224351s) [noise=true , address=77.90.31.14:4664 , piece_id=7SWB2JHTTARLJSPA7EO77NB7DO5WZ3GQDX5NJ4WTEAYIFRTJ5C5Q, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.441455948 (541h47m55.312538983s) [noise=true , address=45.91.116.8:4047 , piece_id=EE6TRGVDYQC6W5MZJNJXVQFSJQNMX763ND7S7HRJM34XF3IBI7JQ, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.441884073 (541h47m55.312967108s) [noise=true , address=145.79.227.5:4223 , piece_id=U3DBMPC2Z6GA3P33ZHOY5BINRJ5HK6RTX7RUO5JLNEHETTA3EAHQ, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.442142815 (541h47m55.31322586s) [noise=true , address=145.79.227.45:4230 , piece_id=7BTO527ZUQPWAT2L6XQZ3X7YWGWYIVBLCAIDXKXHW7WGEDVPAX7A, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.442171991 (541h47m55.313255026s) [noise=true , address=145.79.227.31:4799 , piece_id=CIDWIS626YGAYKOW4RGX222HM73KG7UHR5FVJONVILRLTDDOOZVA, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.442715236 (541h47m55.313798271s) [noise=true , address=145.79.227.36:4519 , piece_id=EGZ4INQBHKSJCIX2RLJOFBNWAGJTCUKWPSTUQ77T5QCPDK2RXKMQ, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.442838690 (541h47m55.313921725s) [noise=true , address=77.90.31.10:4519 , piece_id=TFGFNB2Z6N5OSOUL7L7K4XK32RMBCJLC5HWPAJXJVMGFEYHCVKRA, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.443197033 (541h47m55.314280079s) [noise=true , address=145.79.227.43:4158 , piece_id=3MXPF7UIUVERLTQAJZQPAN2CYRNYMG2EHDLBCANRJTDKEBV6PFZQ, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.443288036 (541h47m55.314371081s) [noise=true , address=145.79.227.51:4409 , piece_id=ODZ2XH2WB55AQFHMKARZMYFIK7IZQXKOOOQGMZAQAXUVJPMZNYFQ, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.443666828 (541h47m55.314749873s) [noise=true , address=45.91.116.6:4975 , piece_id=5OLALQWBBQLJUNAWOASIHPYAYZ4OEPSZXVBJUPCFAEUIWGPAEWFA, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.443698237 (541h47m55.314781272s) [noise=true , address=145.79.227.42:4122 , piece_id=T2OQ6AGNA46LYY23XEXRPMJDWWVC3YWLLKLWZVXMQZV3P5L22AOA, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.444187088 (541h47m55.315270133s) [noise=true , address=145.79.227.37:4567 , piece_id=3RSFUBEETQOZZXOQW6PGWLYXTPEU3JPXO6SF6AF2D5YPD453VIJQ, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.444672212 (541h47m55.315755267s) [noise=true , address=45.91.116.5:4298 , piece_id=ZMIOJI54EILHRWIK3GHFHE6H3Y63KTCNJ3D7J6I76TIAKCNXQDJA, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:55.445185159 (541h47m55.316268204s) [noise=true , address=145.79.227.15:4875 , piece_id=BUXIEMS7O6XW7LNAX2MTJ2KHVYZZHI2FZ2SM6UWU4CRD24YTEZCA, seg_pos={0 0}, upload=475126]: Uploading piece... Dec 28 05:16:57.061329025 (541h47m56.932412071s) [upload=475126]: Next returned nil segment Dec 28 05:16:57.061351698 (541h47m56.932434743s) [upload=475126]: Waiting for error group managing segments... Dec 28 05:16:57.448498348 (541h47m57.319581383s) [noise=true , address=145.79.227.31:4799 , piece_id=CIDWIS626YGAYKOW4RGX222HM73KG7UHR5FVJONVILRLTDDOOZVA, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.453651320 (541h47m57.324734365s) [noise=true , address=145.79.227.42:4122 , piece_id=T2OQ6AGNA46LYY23XEXRPMJDWWVC3YWLLKLWZVXMQZV3P5L22AOA, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.470739488 (541h47m57.341822523s) [noise=true , address=145.79.227.15:4875 , piece_id=BUXIEMS7O6XW7LNAX2MTJ2KHVYZZHI2FZ2SM6UWU4CRD24YTEZCA, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.475963336 (541h47m57.347046381s) [noise=true , address=145.79.227.37:4567 , piece_id=3RSFUBEETQOZZXOQW6PGWLYXTPEU3JPXO6SF6AF2D5YPD453VIJQ, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.694081456 (541h47m57.565164491s) [noise=true , address=45.91.116.7:4010 , piece_id=CW5X37NUMNFTMMP6FIQKNRNZD5RBNM547GCKOBDGAXQ7JKN2CECQ, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.744624128 (541h47m57.615707164s) [noise=true , address=77.90.31.14:4664 , piece_id=7SWB2JHTTARLJSPA7EO77NB7DO5WZ3GQDX5NJ4WTEAYIFRTJ5C5Q, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.765042718 (541h47m57.636125753s) [noise=true , address=45.91.116.8:4047 , piece_id=EE6TRGVDYQC6W5MZJNJXVQFSJQNMX763ND7S7HRJM34XF3IBI7JQ, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.824551132 (541h47m57.695634167s) [noise=true , address=45.91.116.6:4975 , piece_id=5OLALQWBBQLJUNAWOASIHPYAYZ4OEPSZXVBJUPCFAEUIWGPAEWFA, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.831320121 (541h47m57.702403156s) [noise=true , address=145.79.227.46:4603 , piece_id=447I7VFIUL42O7ZLM2LBCTERPXZDAVNHSLEF2TAIYYRHJF2TNK5Q, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.831617848 (541h47m57.702700894s) [noise=true , address=145.79.227.5:4223 , piece_id=U3DBMPC2Z6GA3P33ZHOY5BINRJ5HK6RTX7RUO5JLNEHETTA3EAHQ, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.839020544 (541h47m57.710103579s) [noise=true , address=145.79.227.36:4519 , piece_id=EGZ4INQBHKSJCIX2RLJOFBNWAGJTCUKWPSTUQ77T5QCPDK2RXKMQ, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:57.867273021 (541h47m57.738356066s) [noise=true , address=145.79.227.45:4230 , piece_id=7BTO527ZUQPWAT2L6XQZ3X7YWGWYIVBLCAIDXKXHW7WGEDVPAX7A, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:58.442875250 (541h47m58.313958295s) [noise=true , address=145.79.227.51:4409 , piece_id=ODZ2XH2WB55AQFHMKARZMYFIK7IZQXKOOOQGMZAQAXUVJPMZNYFQ, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:58.442895669 (541h47m58.313978714s) [seg_pos={0 0}, upload=475126]: Segment reached optimal threshold of 13 pieces. Dec 28 05:16:58.443144042 (541h47m58.314227087s) [noise=true , address=45.91.116.5:4298 , piece_id=ZMIOJI54EILHRWIK3GHFHE6H3Y63KTCNJ3D7J6I76TIAKCNXQDJA, seg_pos={0 0}, upload=475126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 05:16:58.443151456 (541h47m58.314234491s) [noise=true , address=45.91.116.5:4298 , piece_id=ZMIOJI54EILHRWIK3GHFHE6H3Y63KTCNJ3D7J6I76TIAKCNXQDJA, seg_pos={0 0}, upload=475126]: Long tail cancellation due to optimal threshold Dec 28 05:16:58.443286222 (541h47m58.314369267s) [noise=true , address=77.90.31.10:4519 , piece_id=TFGFNB2Z6N5OSOUL7L7K4XK32RMBCJLC5HWPAJXJVMGFEYHCVKRA, seg_pos={0 0}, upload=475126]: Done uploading piece. err: <nil> Dec 28 05:16:58.443454954 (541h47m58.314537999s) [noise=true , address=145.79.227.43:4158 , piece_id=3MXPF7UIUVERLTQAJZQPAN2CYRNYMG2EHDLBCANRJTDKEBV6PFZQ, seg_pos={0 0}, upload=475126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 05:16:58.443465434 (541h47m58.314548479s) [noise=true , address=145.79.227.43:4158 , piece_id=3MXPF7UIUVERLTQAJZQPAN2CYRNYMG2EHDLBCANRJTDKEBV6PFZQ, seg_pos={0 0}, upload=475126]: Long tail cancellation due to optimal threshold Dec 28 05:16:58.443488237 (541h47m58.314571282s) [seg_pos={0 0}, upload=475126]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 05:16:58.443564142 (541h47m58.314647187s) [upload=475126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:16:58.443571455 (541h47m58.3146545s) [upload=475126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:16:58.730305032 (541h47m58.601388087s) [upload=475126]: Done uploading segments... Dec 28 05:16:58.730328877 (541h47m58.601411922s) [upload=475126]: Upload finished. err: <nil> Duration: 5.952948955s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)