Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.745485921s FAIL (2025-10-26 09:38:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 5.745514033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.888512022s FAIL (2025-10-26 09:38:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, Piece ID: PR6AHBMQ6ZES2HZCNCLOMCGVBVSNUB3RDZDNL3Z4HVB3REE7BTEQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6PZQAUOOHEYBD554UTVBHKROER3L2OM2DXZT7TMR5FEGVVDD7AXQ) context deadline exceeded ======LOGS====== Oct 26 09:37:34.123071205 (788h6m19.585163678s) [upload=693243]: Starting upload Oct 26 09:37:34.123157710 (788h6m19.585250193s) [upload=693243]: Done starting upload Oct 26 09:37:34.123221812 (788h6m19.585314285s) [upload=693243]: Uploading segments... Oct 26 09:37:34.123433054 (788h6m19.585525527s) [upload=693243]: Got next segment. Inline: false Oct 26 09:37:34.123468111 (788h6m19.585560584s) [upload=693243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 09:37:34.123473151 (788h6m19.585565624s) [upload=693243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 09:37:34.371973433 (788h6m19.834065906s) [seg_pos={0 0}, upload=693243]: Begin upload segment... Oct 26 09:37:34.372032737 (788h6m19.83412521s) [seg_pos={0 0}, upload=693243]: Done begin upload segment. Oct 26 09:37:34.372057634 (788h6m19.834150107s) [noise=true , address=145.79.227.18:4005 , piece_id=QO5MPWZIVIBC2Z4KZXN5WCZT5K5EPWVVQANET4UTIMRSTRXRTI6A, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.372108300 (788h6m19.834200813s) [noise=true , address=145.79.227.3:4151 , piece_id=DEQRVZTIXDG6OKYJRJP7EFRQ6KXOXR62T42WFVK6W6MJTTDN4NAQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.373623074 (788h6m19.835715547s) [noise=true , address=145.79.227.23:4545 , piece_id=CSR54JE45RBS764V42NL6US73UGKU47KFREREDNKZPCV2L2W47NQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.373860316 (788h6m19.835952829s) [noise=true , address=145.79.227.22:4511 , piece_id=TCOMYXXU7CFYAM73AUQ66IOF6XHAHFV6YKPGUEX4BQ3ZAAN7REKA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.374152954 (788h6m19.836245427s) [noise=true , address=145.79.227.17:4970 , piece_id=P6VHRKEVMDQAYGZIK5OOZK7IRHDGG24DD67BGMWKJGHDCPBMB2IA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.374700806 (788h6m19.83679331s) [noise=true , address=45.91.116.4:4937 , piece_id=36AU7KAHTQXPWVVE4PGJGJJQD54V3I7D2JZX3RUDAG76MJSE6N7A, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.375448481 (788h6m19.837540954s) [noise=true , address=77.90.31.18:4805 , piece_id=6PZQAUOOHEYBD554UTVBHKROER3L2OM2DXZT7TMR5FEGVVDD7AXQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.375998859 (788h6m19.838091362s) [noise=true , address=145.79.227.18:4006 , piece_id=OBAUIRAVT5TNOSXNWFIP7NX7LOAKCH5M7MMROPJHFX6YZATNARIQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.377161182 (788h6m19.839253655s) [noise=true , address=145.79.227.25:4619 , piece_id=MFYNCKJTG5NSMCC4IQ6RAZIGRY6LYBJK4ZIC6N5HVVZFQTYXXSTQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.378339846 (788h6m19.840432349s) [noise=true , address=45.91.116.11:4117 , piece_id=VNBEA5VQKB4PRKPMPLB6TCMRQ4EIUICH27HCTP2GUI4NKZZFIVDA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.378778221 (788h6m19.840870704s) [noise=true , address=77.90.31.14:4661 , piece_id=UAO2FLMNOVCG6WAELDJ33XUBXYDCMNOFFL5TXPEC7LY6UBA2SRYA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.379959250 (788h6m19.842051723s) [noise=true , address=77.90.31.16:4733 , piece_id=PR6AHBMQ6ZES2HZCNCLOMCGVBVSNUB3RDZDNL3Z4HVB3REE7BTEQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.380473749 (788h6m19.842566232s) [noise=true , address=145.79.227.6:4257 , piece_id=F3SHTZRLL2GHBRIGXTJZCMBIP2FQTE7CV7HI6BOTPGN66JAKMLEA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.381073312 (788h6m19.843165795s) [noise=true , address=145.79.227.21:4474 , piece_id=NO7ZKII6YPPD3W4J75LG3MQ6JJDPXJSS77SHJKGSQ2QLQIXTPLTQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.383435108 (788h6m19.845527581s) [noise=true , address=145.79.227.21:4475 , piece_id=BQSABKMHRP3H67KYRBK2MKJTHHMCFRJRM23D2WTOUMJU2RMS5GNQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.385259823 (788h6m19.847352306s) [noise=true , address=145.79.227.3:4150 , piece_id=QZYKDSR3KBVVGTFHLD6POVZRX2GV6SQ2INT7D522NJZ5YGHMNCZA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:36.070745218 (788h6m21.532837731s) [upload=693243]: Next returned nil segment Oct 26 09:37:36.070766137 (788h6m21.53285862s) [upload=693243]: Waiting for error group managing segments... Oct 26 09:37:36.406762149 (788h6m21.868854632s) [noise=true , address=145.79.227.18:4005 , piece_id=QO5MPWZIVIBC2Z4KZXN5WCZT5K5EPWVVQANET4UTIMRSTRXRTI6A, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.409600124 (788h6m21.871692597s) [noise=true , address=145.79.227.23:4545 , piece_id=CSR54JE45RBS764V42NL6US73UGKU47KFREREDNKZPCV2L2W47NQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.416939049 (788h6m21.879031522s) [noise=true , address=145.79.227.18:4006 , piece_id=OBAUIRAVT5TNOSXNWFIP7NX7LOAKCH5M7MMROPJHFX6YZATNARIQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.418279591 (788h6m21.880372074s) [noise=true , address=145.79.227.22:4511 , piece_id=TCOMYXXU7CFYAM73AUQ66IOF6XHAHFV6YKPGUEX4BQ3ZAAN7REKA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.426303711 (788h6m21.888396214s) [noise=true , address=145.79.227.6:4257 , piece_id=F3SHTZRLL2GHBRIGXTJZCMBIP2FQTE7CV7HI6BOTPGN66JAKMLEA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.430767821 (788h6m21.892860304s) [noise=true , address=145.79.227.21:4475 , piece_id=BQSABKMHRP3H67KYRBK2MKJTHHMCFRJRM23D2WTOUMJU2RMS5GNQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.431738389 (788h6m21.893830872s) [noise=true , address=145.79.227.25:4619 , piece_id=MFYNCKJTG5NSMCC4IQ6RAZIGRY6LYBJK4ZIC6N5HVVZFQTYXXSTQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.439393947 (788h6m21.90148642s) [noise=true , address=145.79.227.17:4970 , piece_id=P6VHRKEVMDQAYGZIK5OOZK7IRHDGG24DD67BGMWKJGHDCPBMB2IA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.442868813 (788h6m21.904961297s) [noise=true , address=145.79.227.3:4151 , piece_id=DEQRVZTIXDG6OKYJRJP7EFRQ6KXOXR62T42WFVK6W6MJTTDN4NAQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.481214000 (788h6m21.943306473s) [noise=true , address=145.79.227.21:4474 , piece_id=NO7ZKII6YPPD3W4J75LG3MQ6JJDPXJSS77SHJKGSQ2QLQIXTPLTQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.485925793 (788h6m21.948018276s) [noise=true , address=145.79.227.3:4150 , piece_id=QZYKDSR3KBVVGTFHLD6POVZRX2GV6SQ2INT7D522NJZ5YGHMNCZA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.577169889 (788h6m22.039262362s) [noise=true , address=77.90.31.16:4733 , piece_id=PR6AHBMQ6ZES2HZCNCLOMCGVBVSNUB3RDZDNL3Z4HVB3REE7BTEQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.616544336 (788h6m22.078636819s) [noise=true , address=77.90.31.18:4805 , piece_id=6PZQAUOOHEYBD554UTVBHKROER3L2OM2DXZT7TMR5FEGVVDD7AXQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.616576046 (788h6m22.078668519s) [seg_pos={0 0}, upload=693243]: Segment reached optimal threshold of 13 pieces. Oct 26 09:37:36.616738325 (788h6m22.078830798s) [noise=true , address=45.91.116.11:4117 , piece_id=VNBEA5VQKB4PRKPMPLB6TCMRQ4EIUICH27HCTP2GUI4NKZZFIVDA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Oct 26 09:37:36.616745569 (788h6m22.078838052s) [noise=true , address=45.91.116.11:4117 , piece_id=VNBEA5VQKB4PRKPMPLB6TCMRQ4EIUICH27HCTP2GUI4NKZZFIVDA, seg_pos={0 0}, upload=693243]: Long tail cancellation due to optimal threshold Oct 26 09:37:36.616832614 (788h6m22.078925097s) [noise=true , address=45.91.116.4:4937 , piece_id=36AU7KAHTQXPWVVE4PGJGJJQD54V3I7D2JZX3RUDAG76MJSE6N7A, seg_pos={0 0}, upload=693243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 26 09:37:36.616840730 (788h6m22.078933203s) [noise=true , address=45.91.116.4:4937 , piece_id=36AU7KAHTQXPWVVE4PGJGJJQD54V3I7D2JZX3RUDAG76MJSE6N7A, seg_pos={0 0}, upload=693243]: Long tail cancellation due to optimal threshold Oct 26 09:37:36.616859936 (788h6m22.078952419s) [noise=true , address=77.90.31.14:4661 , piece_id=UAO2FLMNOVCG6WAELDJ33XUBXYDCMNOFFL5TXPEC7LY6UBA2SRYA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 26 09:37:36.616882380 (788h6m22.078974863s) [noise=true , address=77.90.31.14:4661 , piece_id=UAO2FLMNOVCG6WAELDJ33XUBXYDCMNOFFL5TXPEC7LY6UBA2SRYA, seg_pos={0 0}, upload=693243]: Long tail cancellation due to optimal threshold Oct 26 09:37:36.616922255 (788h6m22.079014738s) [seg_pos={0 0}, upload=693243]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 09:37:36.617562955 (788h6m22.079655538s) [upload=693243]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:37:36.617579246 (788h6m22.07967173s) [upload=693243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:37:36.914166186 (788h6m22.376258659s) [upload=693243]: Done uploading segments... Oct 26 09:37:36.914194310 (788h6m22.376286783s) [upload=693243]: Upload finished. err: <nil> Duration: 5.888530989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003121595s FAIL (2025-10-26 09:38:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, Piece ID: PR6AHBMQ6ZES2HZCNCLOMCGVBVSNUB3RDZDNL3Z4HVB3REE7BTEQ) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6PZQAUOOHEYBD554UTVBHKROER3L2OM2DXZT7TMR5FEGVVDD7AXQ) context deadline exceeded ======LOGS====== Oct 26 09:37:34.123071205 (788h6m19.585163678s) [upload=693243]: Starting upload Oct 26 09:37:34.123157710 (788h6m19.585250193s) [upload=693243]: Done starting upload Oct 26 09:37:34.123221812 (788h6m19.585314285s) [upload=693243]: Uploading segments... Oct 26 09:37:34.123433054 (788h6m19.585525527s) [upload=693243]: Got next segment. Inline: false Oct 26 09:37:34.123468111 (788h6m19.585560584s) [upload=693243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 09:37:34.123473151 (788h6m19.585565624s) [upload=693243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 09:37:34.371973433 (788h6m19.834065906s) [seg_pos={0 0}, upload=693243]: Begin upload segment... Oct 26 09:37:34.372032737 (788h6m19.83412521s) [seg_pos={0 0}, upload=693243]: Done begin upload segment. Oct 26 09:37:34.372057634 (788h6m19.834150107s) [noise=true , address=145.79.227.18:4005 , piece_id=QO5MPWZIVIBC2Z4KZXN5WCZT5K5EPWVVQANET4UTIMRSTRXRTI6A, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.372108300 (788h6m19.834200813s) [noise=true , address=145.79.227.3:4151 , piece_id=DEQRVZTIXDG6OKYJRJP7EFRQ6KXOXR62T42WFVK6W6MJTTDN4NAQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.373623074 (788h6m19.835715547s) [noise=true , address=145.79.227.23:4545 , piece_id=CSR54JE45RBS764V42NL6US73UGKU47KFREREDNKZPCV2L2W47NQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.373860316 (788h6m19.835952829s) [noise=true , address=145.79.227.22:4511 , piece_id=TCOMYXXU7CFYAM73AUQ66IOF6XHAHFV6YKPGUEX4BQ3ZAAN7REKA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.374152954 (788h6m19.836245427s) [noise=true , address=145.79.227.17:4970 , piece_id=P6VHRKEVMDQAYGZIK5OOZK7IRHDGG24DD67BGMWKJGHDCPBMB2IA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.374700806 (788h6m19.83679331s) [noise=true , address=45.91.116.4:4937 , piece_id=36AU7KAHTQXPWVVE4PGJGJJQD54V3I7D2JZX3RUDAG76MJSE6N7A, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.375448481 (788h6m19.837540954s) [noise=true , address=77.90.31.18:4805 , piece_id=6PZQAUOOHEYBD554UTVBHKROER3L2OM2DXZT7TMR5FEGVVDD7AXQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.375998859 (788h6m19.838091362s) [noise=true , address=145.79.227.18:4006 , piece_id=OBAUIRAVT5TNOSXNWFIP7NX7LOAKCH5M7MMROPJHFX6YZATNARIQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.377161182 (788h6m19.839253655s) [noise=true , address=145.79.227.25:4619 , piece_id=MFYNCKJTG5NSMCC4IQ6RAZIGRY6LYBJK4ZIC6N5HVVZFQTYXXSTQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.378339846 (788h6m19.840432349s) [noise=true , address=45.91.116.11:4117 , piece_id=VNBEA5VQKB4PRKPMPLB6TCMRQ4EIUICH27HCTP2GUI4NKZZFIVDA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.378778221 (788h6m19.840870704s) [noise=true , address=77.90.31.14:4661 , piece_id=UAO2FLMNOVCG6WAELDJ33XUBXYDCMNOFFL5TXPEC7LY6UBA2SRYA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.379959250 (788h6m19.842051723s) [noise=true , address=77.90.31.16:4733 , piece_id=PR6AHBMQ6ZES2HZCNCLOMCGVBVSNUB3RDZDNL3Z4HVB3REE7BTEQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.380473749 (788h6m19.842566232s) [noise=true , address=145.79.227.6:4257 , piece_id=F3SHTZRLL2GHBRIGXTJZCMBIP2FQTE7CV7HI6BOTPGN66JAKMLEA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.381073312 (788h6m19.843165795s) [noise=true , address=145.79.227.21:4474 , piece_id=NO7ZKII6YPPD3W4J75LG3MQ6JJDPXJSS77SHJKGSQ2QLQIXTPLTQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.383435108 (788h6m19.845527581s) [noise=true , address=145.79.227.21:4475 , piece_id=BQSABKMHRP3H67KYRBK2MKJTHHMCFRJRM23D2WTOUMJU2RMS5GNQ, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:34.385259823 (788h6m19.847352306s) [noise=true , address=145.79.227.3:4150 , piece_id=QZYKDSR3KBVVGTFHLD6POVZRX2GV6SQ2INT7D522NJZ5YGHMNCZA, seg_pos={0 0}, upload=693243]: Uploading piece... Oct 26 09:37:36.070745218 (788h6m21.532837731s) [upload=693243]: Next returned nil segment Oct 26 09:37:36.070766137 (788h6m21.53285862s) [upload=693243]: Waiting for error group managing segments... Oct 26 09:37:36.406762149 (788h6m21.868854632s) [noise=true , address=145.79.227.18:4005 , piece_id=QO5MPWZIVIBC2Z4KZXN5WCZT5K5EPWVVQANET4UTIMRSTRXRTI6A, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.409600124 (788h6m21.871692597s) [noise=true , address=145.79.227.23:4545 , piece_id=CSR54JE45RBS764V42NL6US73UGKU47KFREREDNKZPCV2L2W47NQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.416939049 (788h6m21.879031522s) [noise=true , address=145.79.227.18:4006 , piece_id=OBAUIRAVT5TNOSXNWFIP7NX7LOAKCH5M7MMROPJHFX6YZATNARIQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.418279591 (788h6m21.880372074s) [noise=true , address=145.79.227.22:4511 , piece_id=TCOMYXXU7CFYAM73AUQ66IOF6XHAHFV6YKPGUEX4BQ3ZAAN7REKA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.426303711 (788h6m21.888396214s) [noise=true , address=145.79.227.6:4257 , piece_id=F3SHTZRLL2GHBRIGXTJZCMBIP2FQTE7CV7HI6BOTPGN66JAKMLEA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.430767821 (788h6m21.892860304s) [noise=true , address=145.79.227.21:4475 , piece_id=BQSABKMHRP3H67KYRBK2MKJTHHMCFRJRM23D2WTOUMJU2RMS5GNQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.431738389 (788h6m21.893830872s) [noise=true , address=145.79.227.25:4619 , piece_id=MFYNCKJTG5NSMCC4IQ6RAZIGRY6LYBJK4ZIC6N5HVVZFQTYXXSTQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.439393947 (788h6m21.90148642s) [noise=true , address=145.79.227.17:4970 , piece_id=P6VHRKEVMDQAYGZIK5OOZK7IRHDGG24DD67BGMWKJGHDCPBMB2IA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.442868813 (788h6m21.904961297s) [noise=true , address=145.79.227.3:4151 , piece_id=DEQRVZTIXDG6OKYJRJP7EFRQ6KXOXR62T42WFVK6W6MJTTDN4NAQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.481214000 (788h6m21.943306473s) [noise=true , address=145.79.227.21:4474 , piece_id=NO7ZKII6YPPD3W4J75LG3MQ6JJDPXJSS77SHJKGSQ2QLQIXTPLTQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.485925793 (788h6m21.948018276s) [noise=true , address=145.79.227.3:4150 , piece_id=QZYKDSR3KBVVGTFHLD6POVZRX2GV6SQ2INT7D522NJZ5YGHMNCZA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.577169889 (788h6m22.039262362s) [noise=true , address=77.90.31.16:4733 , piece_id=PR6AHBMQ6ZES2HZCNCLOMCGVBVSNUB3RDZDNL3Z4HVB3REE7BTEQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.616544336 (788h6m22.078636819s) [noise=true , address=77.90.31.18:4805 , piece_id=6PZQAUOOHEYBD554UTVBHKROER3L2OM2DXZT7TMR5FEGVVDD7AXQ, seg_pos={0 0}, upload=693243]: Done uploading piece. err: <nil> Oct 26 09:37:36.616576046 (788h6m22.078668519s) [seg_pos={0 0}, upload=693243]: Segment reached optimal threshold of 13 pieces. Oct 26 09:37:36.616738325 (788h6m22.078830798s) [noise=true , address=45.91.116.11:4117 , piece_id=VNBEA5VQKB4PRKPMPLB6TCMRQ4EIUICH27HCTP2GUI4NKZZFIVDA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Oct 26 09:37:36.616745569 (788h6m22.078838052s) [noise=true , address=45.91.116.11:4117 , piece_id=VNBEA5VQKB4PRKPMPLB6TCMRQ4EIUICH27HCTP2GUI4NKZZFIVDA, seg_pos={0 0}, upload=693243]: Long tail cancellation due to optimal threshold Oct 26 09:37:36.616832614 (788h6m22.078925097s) [noise=true , address=45.91.116.4:4937 , piece_id=36AU7KAHTQXPWVVE4PGJGJJQD54V3I7D2JZX3RUDAG76MJSE6N7A, seg_pos={0 0}, upload=693243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 26 09:37:36.616840730 (788h6m22.078933203s) [noise=true , address=45.91.116.4:4937 , piece_id=36AU7KAHTQXPWVVE4PGJGJJQD54V3I7D2JZX3RUDAG76MJSE6N7A, seg_pos={0 0}, upload=693243]: Long tail cancellation due to optimal threshold Oct 26 09:37:36.616859936 (788h6m22.078952419s) [noise=true , address=77.90.31.14:4661 , piece_id=UAO2FLMNOVCG6WAELDJ33XUBXYDCMNOFFL5TXPEC7LY6UBA2SRYA, seg_pos={0 0}, upload=693243]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 26 09:37:36.616882380 (788h6m22.078974863s) [noise=true , address=77.90.31.14:4661 , piece_id=UAO2FLMNOVCG6WAELDJ33XUBXYDCMNOFFL5TXPEC7LY6UBA2SRYA, seg_pos={0 0}, upload=693243]: Long tail cancellation due to optimal threshold Oct 26 09:37:36.616922255 (788h6m22.079014738s) [seg_pos={0 0}, upload=693243]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 09:37:36.617562955 (788h6m22.079655538s) [upload=693243]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:37:36.617579246 (788h6m22.07967173s) [upload=693243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:37:36.914166186 (788h6m22.376258659s) [upload=693243]: Done uploading segments... Oct 26 09:37:36.914194310 (788h6m22.376286783s) [upload=693243]: Upload finished. err: <nil> Duration: 30.003201999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 15.820499431s FAIL (2025-10-26 09:37:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 15.820531412s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 20.597975013s FAIL (2025-10-26 09:37:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 20.598017393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 20.921981657s FAIL (2025-10-26 09:36:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 20.922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 25.250232458s FAIL (2025-10-26 09:35:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 25.250262427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 17.777904801s FAIL (2025-10-26 09:34:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 17.77793613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 11.240967434s FAIL (2025-10-26 09:33:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 11.241003994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.502501544s FAIL (2025-10-26 09:32:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 9.502523595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 23.358808034s FAIL (2025-10-26 09:31:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 23.358837942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 16.779093513s FAIL (2025-10-26 09:30:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 16.779119412s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 28.626174837s FAIL (2025-10-26 09:29:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 28.626209694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.945044427s FAIL (2025-10-26 09:28:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 7.94507727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 17.561435012s FAIL (2025-10-26 09:27:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 17.56149196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 16.895686347s FAIL (2025-10-26 09:26:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 16.895714953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 13.907407363s FAIL (2025-10-26 09:25:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 13.907443102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.589777943s FAIL (2025-10-26 09:24:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 7.589805515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 13.065597989s FAIL (2025-10-26 09:24:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 13.065663863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.668732941s FAIL (2025-10-26 09:23:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 6.668765783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 17.666974042s FAIL (2025-10-26 09:23:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 17.667006563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 18.022958253s FAIL (2025-10-26 09:22:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 18.022976978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 19.033839869s FAIL (2025-10-26 09:21:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 19.033870318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 17.052903016s FAIL (2025-10-26 09:20:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 17.052934705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 20.242766724s FAIL (2025-10-26 09:19:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 20.24279001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 18.232842149s FAIL (2025-10-26 09:19:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 18.232899878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 22.905568287s FAIL (2025-10-26 09:18:42 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 22.905595198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 22.65475849s FAIL (2025-10-26 09:18:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 22.654789709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.881661071s FAIL (2025-10-26 09:17:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 7.881698192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.605771953s FAIL (2025-10-26 09:16:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 9.605854129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 16.286568406s FAIL (2025-10-26 09:15:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 16.28669084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 9.867764945s FAIL (2025-10-26 09:14:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 9.867794512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 11.714461366s FAIL (2025-10-26 09:14:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 11.714501932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 17.585074966s FAIL (2025-10-26 09:13:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 17.585100085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 18.721791899s FAIL (2025-10-26 09:13:09 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 18.721816505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002467381s FAIL (2025-10-26 09:12:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:51.222330643 (787h40m36.684423116s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.222787022 (787h40m36.684879505s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223169571 (787h40m36.685262054s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223453251 (787h40m36.685545734s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.223785975 (787h40m36.685878458s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224102177 (787h40m36.68619466s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224574195 (787h40m36.686666678s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.224872674 (787h40m36.686965157s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225214946 (787h40m36.687307419s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225560494 (787h40m36.687652967s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.225956848 (787h40m36.688049331s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.226364384 (787h40m36.688456867s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227523551 (787h40m36.689616034s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.227745984 (787h40m36.689838467s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.228765657 (787h40m36.69085813s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.229181408 (787h40m36.691273882s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:51.230599488 (787h40m36.692691961s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Uploading piece... Oct 26 09:11:52.727018255 (787h40m38.189110738s) [upload=692861]: Next returned nil segment Oct 26 09:11:52.727082728 (787h40m38.189175201s) [upload=692861]: Waiting for error group managing segments... Oct 26 09:11:53.134330412 (787h40m38.596422885s) [noise=true , address=145.79.227.11:4737 , piece_id=F6ANXEFIQ2V6IJFFNSRB56MHQZF5JCLQQBGCNZWQKGRDCF7SUKSQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.141654859 (787h40m38.603747342s) [noise=true , address=145.79.227.5:4238 , piece_id=WG33KN3SBYNJLES3DTLVRKNJUFOYRVD6WI2SKUVVRXUKURVQX7CQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.145129877 (787h40m38.60722235s) [noise=true , address=145.79.227.7:4324 , piece_id=TOAE5LURPGZBD2OJUCMMW2WQSXFZPVOREP7HX2XD63667URT5SDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.161148489 (787h40m38.623241012s) [noise=true , address=145.79.227.16:4931 , piece_id=VY2IF6TONVU7TBBWW5DBSEQMWV64FSTI54EDN6I7QCRHN3GHOUDA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.205126371 (787h40m38.667218854s) [noise=true , address=145.79.227.38:4002 , piece_id=2WN2ARMGOQLFHV24TBIOQWJHKV6FJZ6O6FBTASAEJKMJHW5EV3UQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.244490708 (787h40m38.706583181s) [noise=true , address=45.91.116.10:4092 , piece_id=7GVTGRP5BHVAOPGYVZXDORWLTPV37WJUQ5SV4Y7TFHC44YIBD6PA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.245945258 (787h40m38.708037731s) [noise=true , address=77.90.31.19:4863 , piece_id=L22445LDWTRC5SB7UONF5TKSNPH57XN4DIBH3SMXRSRIIKNZZIWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.305496434 (787h40m38.767588927s) [noise=true , address=45.91.116.12:4229 , piece_id=3U7JDSZN3BARD4KIVTEEABPQKCOISXJNHQUH76H7Z6YJDVXFE5VA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.319540126 (787h40m38.781632599s) [noise=true , address=77.90.31.21:4892 , piece_id=2UIEK666YKUL4TLMGCZNSRVMH2TZQNQZGILY4P3U6F6YJQBM26JQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.329271826 (787h40m38.791364299s) [noise=true , address=45.91.116.14:4223 , piece_id=LXBPNQC6U666UP5G3QU5TDCULLERPFCOYMBR5ZOAOMKFW3IJZFKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.332343686 (787h40m38.79443617s) [noise=true , address=45.91.116.9:4280 , piece_id=4MKC26EJSOVQO6B63AIATIPSNKFSLEPOME6RSDEPM35INF2OFKKA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.342035802 (787h40m38.804128276s) [noise=true , address=77.90.31.12:4622 , piece_id=UL26HAE7NMBEOIBGHAD7MMIERZ6O3DP56YDDQU3GRTTCDEO5XIGQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.373785539 (787h40m38.835878022s) [noise=true , address=77.90.31.18:4823 , piece_id=YXHIIYK733X3ZG5L37ZFXXTLFZEKFTBN6375DK7S433WIQI63PZA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390469179 (787h40m38.852561662s) [noise=true , address=77.90.31.14:4670 , piece_id=OVXQ4YW4O34ZPKW235WJ5NMPMW5SCVNFDYXTUIFOIREYXLTFYMJA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: <nil> Oct 26 09:11:53.390500137 (787h40m38.85259261s) [seg_pos={0 0}, upload=692861]: Segment reached optimal threshold of 14 pieces. Oct 26 09:11:53.390630505 (787h40m38.852722988s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b9DdnzjhU7zj2RSCba3ZWS7AZbZZ4YmDDF4QRq6dZC6YfskNz): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390639823 (787h40m38.852732296s) [noise=true , address=45.91.116.7:4044 , piece_id=YFWXOA5M6FKTZSDGRI6EZ7YBUT5X4WTBCXYI7HWOIT7AMLR5UNWA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390699396 (787h40m38.852791869s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390705007 (787h40m38.85279748s) [noise=true , address=77.90.31.13:4635 , piece_id=KJBCXXOHTIHCQQACD26L52WFYLQZPMLTW3MN2QA7HM5R46DUSNYA, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390724324 (787h40m38.852816797s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390785290 (787h40m38.852877764s) [noise=true , address=77.90.31.16:4753 , piece_id=6HIZXMBAANFRBPEJYI6XMEO33QYG6KSCI3KCYNK37DBC6AW7G73A, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390800329 (787h40m38.852892812s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Oct 26 09:11:53.390807993 (787h40m38.852900466s) [noise=true , address=77.90.31.15:4701 , piece_id=GRKLC6CH5BQNYTOIL6JDLO6DDYB26CLXLKMLV3EJLE2IRJO3PFEQ, seg_pos={0 0}, upload=692861]: Long tail cancellation due to optimal threshold Oct 26 09:11:53.390831508 (787h40m38.852923981s) [seg_pos={0 0}, upload=692861]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 26 09:11:53.390904788 (787h40m38.852997261s) [upload=692861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 09:11:53.390909757 (787h40m38.85300224s) [upload=692861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 09:11:53.689354952 (787h40m39.151447445s) [upload=692861]: Done uploading segments... Oct 26 09:11:53.689380491 (787h40m39.151472974s) [upload=692861]: Upload finished. err: <nil> Duration: 1m0.002489532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.797336996s FAIL (2025-10-26 05:45:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.79747081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.941594689s FAIL (2025-10-26 05:44:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.941625017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.980103778s FAIL (2025-10-26 05:43:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.980149937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.961562407s FAIL (2025-10-26 05:42:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.961594227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.883320935s FAIL (2025-10-26 05:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.883354868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.95695135s FAIL (2025-10-26 05:42:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.956971699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.830827728s FAIL (2025-10-26 05:41:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.830868386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.915958885s FAIL (2025-10-26 05:41:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.915993473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.766907731s FAIL (2025-10-26 05:40:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 5.766941355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.45899527s FAIL (2025-10-26 05:39:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 7.45902641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.049672248s FAIL (2025-10-26 05:39:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 6.049709558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.014270435s FAIL (2025-10-26 05:37:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 6.014310802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.053089932s FAIL (2025-10-26 05:36:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 6.053123576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.005851895s FAIL (2025-10-26 05:35:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ) context deadline exceeded; piecestore: (Node ID: 1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, Piece ID: QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA) context deadline exceeded ======LOGS====== Oct 26 05:01:16.395079546 (783h30m1.857172019s) [upload=689172]: Starting upload Oct 26 05:01:16.395176260 (783h30m1.857268743s) [upload=689172]: Done starting upload Oct 26 05:01:16.395304083 (783h30m1.857396566s) [upload=689172]: Uploading segments... Oct 26 05:01:16.395473306 (783h30m1.857565809s) [upload=689172]: Got next segment. Inline: false Oct 26 05:01:16.395514314 (783h30m1.857606787s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 05:01:16.395520565 (783h30m1.857613048s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 05:01:16.758259781 (783h30m2.220352264s) [seg_pos={0 0}, upload=689172]: Begin upload segment... Oct 26 05:01:16.758342769 (783h30m2.220435262s) [seg_pos={0 0}, upload=689172]: Done begin upload segment. Oct 26 05:01:16.758381082 (783h30m2.220473555s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.758429484 (783h30m2.220521967s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759193259 (783h30m2.221285742s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.759663444 (783h30m2.221755927s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760135613 (783h30m2.222228096s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.760600829 (783h30m2.222693312s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761462940 (783h30m2.223555423s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.761480734 (783h30m2.223573227s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.764118006 (783h30m2.226210479s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765006799 (783h30m2.227099292s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.765887155 (783h30m2.227979638s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767090998 (783h30m2.229183491s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.767644632 (783h30m2.229737125s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.768664273 (783h30m2.230756756s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769214892 (783h30m2.231307375s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:16.769959920 (783h30m2.232052403s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Uploading piece... Oct 26 05:01:18.487250724 (783h30m3.949343207s) [upload=689172]: Next returned nil segment Oct 26 05:01:18.487274199 (783h30m3.949366682s) [upload=689172]: Waiting for error group managing segments... Oct 26 05:01:18.777449197 (783h30m4.23954167s) [noise=true , address=145.79.227.17:4972 , piece_id=EHHLFOGHENQ2PYDEI7OP7UD6LJDHLRJ542IXZ4SIMVAHH3ZPEIQQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.778032639 (783h30m4.240125112s) [noise=true , address=145.79.227.18:4005 , piece_id=PO42XGZSPUP7FG5W3NKGRYHSR6SB7CXUSQAANR3NEE2X52GPQJFQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.783633053 (783h30m4.245725526s) [noise=true , address=145.79.227.14:4837 , piece_id=M6CBT3GMCCVOL32GMQGQSWJGXQQJZX4N74ED4BHPFVTMARFNZM3A, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.785216378 (783h30m4.247308851s) [noise=true , address=145.79.227.10:4401 , piece_id=JCRGFLRUG6I5MH6HETA4DAGG6EBHI5IPR7R5FUIS4YKE7UJDXCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.810281477 (783h30m4.27237396s) [noise=true , address=145.79.227.3:4151 , piece_id=XDC6U7GSA3J7GV5QFX5MAZR52J2MXIHQXUJEXFF2MRMY3UXSMZOA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.825235833 (783h30m4.287328316s) [noise=true , address=145.79.227.13:4803 , piece_id=4EIFOWNZV4NXTGWIL2TOPR5TPW2HDLEPADNC65RWHWKVOCZ76RVQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.848823097 (783h30m4.31091558s) [noise=true , address=145.79.227.16:4910 , piece_id=U4BONTUQU54M2T4HT4YY24S6IS5HAEUMLGQT643NSAMA6OMTKFYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.871969293 (783h30m4.334061766s) [noise=true , address=145.79.227.20:4439 , piece_id=YXP556UP6YFEE6P7N2SM4AJVPUWXNALX4JII5Q5SYONOY6QDMCYA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:18.964001690 (783h30m4.426094163s) [noise=true , address=77.90.31.12:4589 , piece_id=RTOHPQHCUCSOPFNCQ6QGVCWLOLKAKDTELZJ7QONCXIV4NY43CJ5Q, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.046379172 (783h30m4.508471645s) [noise=true , address=77.90.31.13:4627 , piece_id=2YFK7MHUENGJM6CUGRWKFOTPZZSOLMTCS77VSHLCM5FZZTJEG6KQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.060947854 (783h30m4.523040337s) [noise=true , address=45.91.116.6:4974 , piece_id=UMCAZVNB63SPD5JXOKSRPLXJ7YXD5WUZCJ7N6LPGRDTUTSZH3UIQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.087740683 (783h30m4.549833156s) [noise=true , address=45.91.116.10:4083 , piece_id=ZFU2BAYP72QFEQJNWDZDEXN4KDHWTWMLZL66N626TLOK2NPIXLRQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089823429 (783h30m4.551915912s) [noise=true , address=77.90.31.16:4734 , piece_id=QZINDZJGKE7KMV4FF5PLZMJSNVRN5DNF2NO4RBX2IQPCTGSCY2HA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: <nil> Oct 26 05:01:19.089840592 (783h30m4.551933065s) [seg_pos={0 0}, upload=689172]: Segment reached optimal threshold of 13 pieces. Oct 26 05:01:19.089951593 (783h30m4.552044076s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.089958065 (783h30m4.552050538s) [noise=true , address=77.90.31.15:4697 , piece_id=6YGVQI4LSGQGF22SDNOSWXJ3CYXMOLKIOYVX7D4G75KCYADDEHYQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090006127 (783h30m4.55209861s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090034020 (783h30m4.552126503s) [noise=true , address=45.91.116.10:4082 , piece_id=DQDZ6NCSXI3JIBXPAPKBQRNY5L6EYPVTXBN5DAJ6BGQLQHIU4HNA, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090050121 (783h30m4.552142594s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE): protocol: expected piece hash; context canceled; EOF Oct 26 05:01:19.090059608 (783h30m4.552152091s) [noise=true , address=77.90.31.17:4772 , piece_id=NHMTMKS3KHJRD4UM7ZW2E6MDDRYJTSVK2WVZPQNF5QDSNU7GF4NQ, seg_pos={0 0}, upload=689172]: Long tail cancellation due to optimal threshold Oct 26 05:01:19.090089205 (783h30m4.552181688s) [seg_pos={0 0}, upload=689172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 05:01:19.090160911 (783h30m4.552253384s) [upload=689172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 05:01:19.090165881 (783h30m4.552258364s) [upload=689172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 05:01:19.403485061 (783h30m4.865577544s) [upload=689172]: Done uploading segments... Oct 26 05:01:19.403506602 (783h30m4.865599075s) [upload=689172]: Upload finished. err: <nil> Duration: 6.005885879s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)